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(54) Data processing apparatus and method 

(57) In decoding code data encoded in object units, 
decoders corresponding to the number of objects are 
needed. However, it is impossible to always provide a 
sufficient number of decoder. Accordingly, when code 
data 8 is decoded, an object combiner 43 refers to the 
number s of objects included in the code data 8. 



detected by an object counter 41, and the number d of 
object decoders, detected by an object decoder counter 
42. rf s>d holds, the object combiner 43 regulates the 
number of the objects of the input code data 8 to d. 
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Description 

BACKGROUND OF THE INVENTION 



5 FIELD OF THE INVENTION 

[0001] The present invention relates to decoding apparatus and method and. more particularly, to data processing 
apparatus and method which decode code encoded in image object units. 

[0002] Further, the present invention relates to data processing apparatus and method which process a data array 
10 constructing an image with a plurality of coded image objects. 

DESCRIPTION OF RELATED ART 

[0003] In recent years, with advancement in image encoding techniques and progress of computer capabilities, an 
is encoding method to separate an image into objects and encode by each object has been proposed. The image encod- 
ing in object units enables optimum encoding by each object, thus improving the coding efficiency. At the same time, a 
function to generate a new image by editing the objects within the image can be obtained. For example, in the technol- 
ogy of still image, a method to separate an image into "character", "line", "frame", "image", "table" and "background-, 
and perform optimum encoding on the respective areas, such as the ACBIS method (by Maeda, and Yoshida in "The 
20 1996 Institute of Electronics. Information and Communication Engineers General Conference D-292") has been pro- 
posed. According to this method, the JBIG (Joint BMevel Image Group) encoding as a binary-image encoding method 
is performed on the "character", "line", "frame" and "table" areas, and in the "background" area, its representative value 
is encoded. 

[0004] Further, in a moving image, a method to perform encoding in object units has been studied as an international 
25 standard method, MPEG4 (Moving Picture Experts Group phase 4) (Eto. "MPEG4 Standardization" (The Journal of The 
Institute of Image Electronics Engineers of Japan, vol. 25, No. 3, 1996. pp. 223-228). Rg. 1 shows an example of a 
frame of a moving image to be encoded by the MPEG4 coding. In Fig. 1 . a frame 20 comprises four objects as shown 
in Fig. 2, i.e., a background object 28, an object 21 representing a helicopter, an object 22 representing a train, and an 
object 23 representing a car. To indicate the shapes of the objects except the background, each object is masked such 
30 that a black part of a rectangular area surrounding the object is an "outer area", and a white part is an "inner area" (24 
to 26 in Fig. 2), and by this masking, an arbitrary shaped object can be handled. 

[0005] Fig. 3 shows a construction for coding in object units. An input image 1 is inputted into an object segmenter 2, 
and is separated into respective objects. For example, the image in Fig. 1 is separated by the object segmenter 2 into 
the objects 28. 21 , 22 and 23, and the objects are independently encoded. That is. an object encoder 3 encodes the 
35 object 28; an object encoder 4, the object 21 ; an object encoder 5, the object 22; and an object encoder 6, the object 
23. A multiplexer 7 multiplexes code data outputted from the object encoders 3 to 6, and outputs the multiplexed data 
as code data 8. 

[0006] Fig. 4 shows a construction for decoding an image encoded in object units. The code data 8 is inputted into a 
demultiplexer 9, and separated into code data corresponding to the respective objects. The separated code data are 
40 independently decoded. That is, an object decoder 10 decodes the object 28; an object decoder 1 1, the object 21; an 
object decoder 12, the object 22; and an object decoder 1 3, the object 23. An object compositer 14 arranges image data 
outputted from the object decoders 10 to 1 3 in proper positions for the respective objects, thus composes them as one 
image, and outputs the image data as a reproduced image 1 5. 

[0007] In moving image coding represented by the MPEG2 (Moving Picture Experts Group phase 2) standard, coding 
45 is made in frame or field units. To realize reuse or editing of contents (person, building, voice, sound, background and 
the like) constructing a video image and audio data of a moving image, the MPEG4 standard is characterized by han- 
dling video data and audio data as objects. Further, objects included in a video image area independently encoded, and 
the objects are independently handled. 

[0008] Fig. 25 shows an example of the structure of object code data. The moving image code data based on the 
so MPEG4 standard has a hierarchical structure, from the point of improvement in coding efficiency and editing operability. 
As shown in Fig. 25. the head of code data has a visual_object_sequence_start_code (VOSSC in Fig. 25) for identifi- 
cation. Then, code data of respective visual objects follows, and visual_object_sequence_end_code (VOSEC in Rg. 
25) indicative of the rear end of the code data is positioned at the end. As well as obtained moving images, computer 
graphics (CG) data and the like are defined as visual objects. 
55 [0009] The visual object data has visual_object_start_oode (Visual Object SC in Rg. 25) for identification at its 
header, then profile_and_leveyndication (PLI in Fig. 25) indicative of an encoding level. Then, information on visual 
objects, is_visual-object_identifier (IVOI in Fig. 25). visuaLobject_varid (VOVID in Fig. 25), visua!_object_priority 
(VOPRI in Fig. 25), visual_object_type (VOTYPE in Fig. 25) and the like follow. These data construct header information 
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of the visual object. "VOTYPE" has a value "0001* if the image is a moving image obtained by image pickup. Then, 
video object (VO) data as a cluster of moving image code data follows. 

[0010] The VO data is code data indicative of each object. The VO data has video_object_start_code (VOSC in Fig. 

25) for identification at its header, further, the VO data has video object layer data (VOL data in Fig. 25) to realize seal- 
5 ability. The VOL data has video_objectjayer_start_code (VOLSC in Fig. 25) and video object plane data (VOP data in 

Fig. 25) corresponding to one frame of moving image. The VOL data has video_object_layer_width (VOL_width in Fig. 

25) and video_object_layer_h eight (VOL_h eight in Fig. 25) indicative of size, at its header. Also, the VOP data has 

video_object_ptane_width (VOP_width in Fig. 25) and video_object_plane_height (VOP_height in Fig. 25) indicative of 

size, at its header. Further, the header of the VOL data has bit_rate code indicative of bit rate. 
io [0011] Note that in each layer of the code data structure, data of an arbitrary length which starts with 

user_data_start_code can be inserted by a user. The user data is distinguished from the code data by recognition of 

start code VOSC and VOLSC or VOPSC following the user data. 

[001 2] Further, arrangement information, which is information to arrange the respective objects on the decoding side, 
is called a system code. In the system code, similar to VRML (Virtual Reality Markup Language) as a CG language, 
is information describing arrangement of divided objects, reproduction timing or the like is encoded. The system code 
describes the relation among the respective objects with conception of nodes. Hereinbelow, the nodes will be specif i* 
cally described with reference to Figs. 26 and 27. 

[001 3] Fig. 26 is an example of an image constructed with a plurality of objects. This image comprises a Background 
object 2000, a Balloon object 2001 , a Bird object 2002, a Jet object 2003, a Car object 2004, a Woman object 2005 and 

so a Man object 2006, respectively representing background, a balloon, a bird, an airplane, a car, a woman and a man. 
[0014] Fig. 27 shows a node tree in the image in Fig. 26. The entire image is represented by a Scene node. The 
Scene node is connected to the Background object 2000, the Car object 2004, and a People node indicative of people 
and a Fly node indicative of things flying in the sky. Further, the People node is connected to the Woman object 2005 
and the Man object 2006. The Fly node is connected to the Balloon object 2001 , the Bird object 2002 and the Jet object 

25 2003. The relation among the objects is described in the data of the system code. 

[001 5] In this manner, according to the MPEG4 standard, by handling objects in a moving image independently, the 
decoding side can freely arrange various objects. Further, in broadcasting companies, contents producing companies 
and the like, by generating code data of objects beforehand, a very large number of moving image data can be gener- 
ated from limited contents. 

30 [0016] However, the above-described techniques have the following problems. To decode respective objects inde- 
pendently, decoders corresponding to the number of separated objects are required. However, on the decoding side, it 
is impossible to prepare an arbitrary number of decoders. Accordingly, the number of independently encoded objects 
may be larger than the number of prepared decoders. The decoding apparatus as shown in Fig. 5 has three object 
decoders. A demultiplexer 9 allocates the object decoders to the code data 8 in input order. If the code data 8 includes 

35 four objects, the demultiplexer 9 allocates the object 28 to the object decoder 10, the object 21 , to the object decoder 
11, and the object 22. to the object decoder 12. However, regarding the object 23. as there is no available object 
decoder, the object 23 is not decoded. Accordingly, in an image obtained by decoding the objects and synthesizing 
them, the object 23 is omitted, as in a frame 38 in Fig. 6. 

[0017] That is, in the coding based on the MPEG4 standard, as an unspecified number of objects are handled, the 
40 number of decoding means to decode all the objects cannot be determined especially on the decoding side, accord- 
ingly, it is very difficult to construct an apparatus or system. For this reason, in the standardized MPEG4 coding, to 
determine the specifications upon designing of code data and encoder/decoder, the concepts of profile and level are 
defined and the number of objects and the upper limit value of bit rate are provided as coding specifications. Fig. 28 
shows an example of a profile table defining the number of objects and the bit rate upper limits of profiles and levels. 
45 [0018] In the MPEG4 standard, a coding tool differs in accordance with profile. Further, as shown in Fig. 28. the 
amount of code data of handled image is determined stepwisely in accordance with level. Note that the maximum 
number of objects to be handled and the maximum bit rate value are upper limits in the coding specifications, and all 
the values are included in the coding specifications as long as they are less than the above maximum values. For exam- 
ple, in a case where a coding tool is available in a Core profile, the number of objects is six, and coding is performed at 
so a bit rate of 300 kbps, the code data and the coding tool correspond to level 2 (Core profile and level 2). 

[001 9] The above-described profile and level are indicated in the P LI in a bit stream of MPEG4 code data as shown 
in Fig. 25. Accordingly, a decoder which decodes a bit stream of MPEG code data can determine whether or not decod- 
ing is possible by referring to the PLI. The decoding is impossible in the following case. 

[0020] For example, a decoder corresponding to of Core profile and level 1 cannot decode code data of Core profile 
55 and level 2 since the maximum bit rate of Core profile and level 2 is 2000 kbps. far higher than 384 kbps as the maxi- 
mum bit rate of Core profile and level 1. 

[0021] Further, in an image including four objects, by synthesizing two code data of Simple profile and level 1, two 
code data of Simple profile and level 2 can be generated. However, as the maximum number of objects of Simple profile 
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and level 2 is 4, code data which cannot belong to any profile or level of the MPEG4 standard is generated. Accordingly, 
such coded data cannot be decoded. 

[0022J Further, for example, if a new bit stream is generated by multiplexing two code data of Simple profile, with bit 
rates 48 kbps and 8 kbps, of two images respectively including two objects, the bit rate of the new bit stream may be 
s over 64 kbps. In this case, the level of the code data must be raised to level 2, and it cannot be decoded by a decoder 
of Simple profile and level 1. 

[0023] That is, if the coding specifications (profile and level) of a decoder do not sufficiently cover the coding specifi- 
cations (profile and level) of code data, the decoder cannot decode the code data. 

[0024] This problem becomes especially outstanding upon synthesizing a plurality of images. For example, when a 
io plurality of code data, respectively decodable by a decoder are synthesized, occasionally the decoder cannot decode 
the synthesized code data. Further, if the synthesized code data does not correspond to any of MPEG4 Profiles and 
levels, it cannot be decoded by a decoder based on the MPEG4 standard. 

SUMMARY OF THE INVENTION 

15 

[0025] The present invention has been made to solve the above-described problems, and has as a concern to provide 
data processing apparatus and method which decode all the image objects even if the number of decoders is limited. 
[0026] According to the present invention there is provided a data processing apparatus having decoding means for 
decoding code encoded in image object units, said apparatus comprising: detection means for detecting the number of 
20 objects included in input code and the number of objects decodable by said decoding means; and control means for 
controlling the number of objects of the input code, based on the number of objects and the number of decodable 
objects detected by said detection means. 

[0027] Further, another concern of the present invention is to provide data processing apparatus and method which 
decode coded still image and/or moving image without degrading the image quality even if the number of decoders is 
25 limited. 

[0028] According to a feature of the present invention, the above-described apparatus further comprises; extraction 
means for extracting location information of the objects included in said code; and combining means for combining code 
of a plurality of objects, based on an instruction from said control means and the location information extracted by said 
extraction means. 

30 [0029] Further the above-described apparatus may comprise: extraction means for extracting motion information 
indicative of motions of the objects included in said code; and combining means for combining a plurality of objects 
based on an instruction from said control means and the motion information extracted by said extraction means. 
[0030] Further, another concern of the present invention is to provide data processing apparatus and method which 
decode code data, encoded by each of plural image objects, with decoders of arbitrary coding specifications. 

35 [0031] Further, another concern of the present invention is to provide data processing apparatus and method which 
control the number of objects included in code data. 

[0032] According to an aspect of the present invention, there is provided a data processing apparatus for processing 
a data array to reproduce an image with a plurality of coded image objects, said apparatus comprising: detection means 
for detecting the number of image objects included in said data array; and control means for controlling the number of 
40 image objects included in said data array based on the number of image objects detected by said detection means. 
[0033] Further, another concern of the present invention is to provide data processing apparatus and method which 
synthesize a plurality of code data, encoded by each of plural image objects, to obtain one code data based on a pre- 
determined coding standard. 

[0034] According to another aspect of the present invention there is provided a data processing apparatus compris- 
es ing: input means for inputting a plurality of image data to construct one frame, wherein said image data respectively 
including N image objects, where N>1 holds; and generation means for generating image data having M image objects, 
where M>1 holds, constructing said one frame, by integrating at least a part of said N image objects based on additional 
information indicative of relation among the image objects. 

[0035] Further, another concern of the present invention is to provide data processing apparatus and method which 

so decode synthesized code data with decoders of arbitrary coding specifications. 

[0036] Further, another concern of the present invention is to provide data processing apparatus and method which 
control the number of objects included in code data and/or the information amount of the code data. 
[0037] According to another aspect of the present invention, there is provided a data processing apparatus for 
processing a data array to reproduce one frame image with a plurality of coded image objects, said apparatus compris- 

55 ing: input means for inputting a plurality of data arrays; instruction means for instructing synthesizing of a plurality of 
data arrays inputted by said input means; designation means for designating coding specifications of a processed data 
array; control means for controlling information amounts of the plurality of data arrays inputted by said input means, 
based on the coding specifications designated by said designation means; and synthesizing means for synthesizing the 
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plurality of data arrays with information amounts controlled by said control means, based on the coding specifications 
designated by said designation means.. 

[0038] Other features and advantages of the present invention will be apparent from the following description taken 
in conjunction with the accompanying drawings, in which like reference characters designate the same name or similar 
5 parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0039] The accompanying drawings, which are incorporated in and constitute a part of the specifications, illustrate 
w embodiments of the invention and, together with the description, serve to explain the principles of the invention. 

Fig. 1 is an example of the image processed by the MPEG4 coding; 
Fig. 2 is an explanatory view showing the objects of the image in Fig. 1 ; 
Fig. 3 is a block diagram showing the construction for coding in object units; 
is Fig. 4 is a block diagram showing the construction for decoding an image encoded in object units; 
Fig. 5 is a block diagram showing the construction for decoding an image encoded in object units; 
Fig. 6 is an example of decoded image where an object is omitted; 

Fig. 7 is a block diagram showing the construction of a decoding apparatus according to the present invention; 
Fig. 8 is an example of 1 -frame code data; 
20 Fig. 9 is an example of synthesized code data; 

Fig. 10 is a block diagram showing the construction of an object combiner according to a first embodiment of the 
present invention; 

Figs. 1 1 A and 1 1 B are examples of object combining; 
Fig. 12 is an example of 1-frame code data to be motion -compensated; 
25 Fig. 1 3 is an example of synthesized code data; 

Figs. 1 4A to 1 4C are examples of objects of a still image and combined objects; 
Fig. 15 is an example of 1 frame of a moving image; 

Fig. 16 is a block diagram showing the construction of the object combiner according to a second embodiment of 
the present invention; 
30 Figs. 1 7A and 1 7B are examples of combined objects; 

Fig. 18 is an example of code data including combined objects; 

Fig. 19 is a block diagram showing the construction of the object combiner according to a third embodiment of the 
present invention; 

Fig. 20 is an example of input code data; 
35 Fig. 2 1 is an example of processed code data; 

Fig. 22 is a block diagram showing the construction of the object combiner according to a fourth embodiment of the 
present invention; 

Fig. 23 is a block diagram showing the construction of the object combiner according to a modification; • 
Fig. 24 is a block diagram showing the construction of the object combiner according to another modification; 
40 Fig. 25 is an example of the structure of object code data; 

Fig. 26 is an example of the image constructed with a plurality of objects; 
Fig. 27 is an example of a node tree in the image in Fig. 26; : i< 

Fig. 28 is an example of the profile table defining the number of objects and the bit rate upper limits by profile and 
level; 

45 Fig. 29 is a block diagram showing the construction of a moving image processing apparatus according to a fifth 
embodiment of the present invention; 

Fig. 30 is a block diagram showing the construction of a profile and level regulator according to the fifth embodi- 
ment; 

Figs. 31 A and 31 B are examples of the structure of code data of moving image; 
so Fig. 32 is a block diagram showing the construction of the profile and level regulator according to a sixth embodi- 
ment of the present invention; 

Fig. 33 is a block diagram showing the construction of the profile and level regulator according to a seventh embod- 
iment of the present invention; 

Fig. 34 is a block diagram showing the construction of an object integrator according to the seventh embodiment; 
55 Fig. 35 is an example of the structure of synthesized code data according to the seventh embodiment: 

Fig. 36 is a block diagram showing the construction of the object integrator according to a modification of the sev- 
enth embodiment; 

Fig. 37 is an example of synthesized color image information according to the seventh embodiment; 
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Fig. 38 is an example of synthesized mask information according to the seventh embodiment; 

Fig. 39 is a block diagram showing the construction of the object integrator according to an eighth embodiment of 

the present invention; 

Fig. 40 is an-example of a slice structure of color image information according to the eighth embodiment; 
Fig. 41 is a block diagram showing the construction of the profile and level regulator according to a ninth embodi- 
ment of the present invention; 

Fig. 42 is an example of the structure of synthesized moving image code data according to the ninth embodiment; 
Fig. 43 is an example of the construction of an image represented by code data; 
Fig. 44 is an example of the construction of an image represented by code data; 

Fig. 45 is a block diagram showing the construction of the moving image processing apparatus according to a tenth 

embodiment of the present invention; 

Figs. 46A to 46D are examples of images to be synthesized; 

Fig. 47 is a block diagram showing the construction of an image editing unit according to the tenth embodiment; 
Fig. 48 is an example of a synthesized image; 

Fig. 49 is a block diagram showing the detailed construction of a header processor; 

Figs. 50A to 50E are examples of code data of images to be synthesized and of a synthesized image; 

Fig. 51 is a flowchart showing image processing according to the tenth embodiment; 

Fig. 52 is a block diagram showing the construction of the image editing unit according to an eleventh embodiment 
of the present invention; 

Figs. 53 A to 53 D are examples of node trees showing the relation among respective objects; 
Fig. 54 is a block diagram showing the construction of a coding length regulator; 

Figs. 55 and 56 are block diagrams showing the constructions of the code length regulator according to modifica- 
tions of the eleventh embodiment; and 
Fig. 57 is an example of code data of a synthesized image. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0040] Preferred embodiments of the present invention will now be described in detail in accordance with the accom- 
panying drawings. 

First Embodiment 

[Construction] 

[0041 ] Fig. 7 is a block diagram showing the construction of a decoding apparatus according to the present invention. 
Note that elements approximately corresponding to those in Figs. 3 and 5 have the same reference numerals and 
detailed explanations of the elements will be omitted. 

[0042] In Fig. 7, an object number regulator 40 includes an object counter 41 which counts the number s of objects, 
an object decoder counter 42 which counts the number d of object decoders, and an object combiner 43 which com- 
bines the plurality of objects included in the code data 8. Numeral 50 denotes a storage device comprising a magnetic 
disk or the like. 

[0043] The code data 8 inputted into the decoding apparatus is subjected to coding by an encoder as shown in Fig. 
3, for example. The code data 8 includes four objects at the maximum. Hereinbelow. description will be made using a 
moving image frame as shown in Fig 1 as an original image. 

[0044] The code data 8 is inputted into the object number regulator 40 by each frame. When code data of a frame has 
been inputted, the object counter 41 counts the numbers of objects included in the code data. 
[0045] Fig. 8 is an example of 1 -frame code data. The code data has "Header" indicative of the attribute of the frame 
at its head, next, code data indicative of background object (Object 0 in Fig. 8). Then, code data of the respective 
objects, i.e., code data of the object 21 (object 1), the object 22 (Object 2) and the Object 23 (Object 3) follow. The code 
data of each object comprises Start code (SC) indicative of the header of the object. Location (Loc) code indicative of 
location of the object Size code indicative of the size of the object. Shape code indicative of the shape of the object, 
and Texture code indicative of the object itself. 

[0046] In the following description, the Shape code is binary-encoded by MR coding, and the Texture code is encoded 
by block-coding. Note that block encoding is dividing an object into. e.g.. 8x8 pixel blocks, then performing discrete 
cosine transformation (DCT) on each block, and quantizing and encoding the obtained conversion coefficients (DCT 
coefficients), such as JPEG coding. Fig. 8 shows the Texture code of the Object 1. The Texture code is a set of block- 
based code, DCT-COEFs. The code DCT-COEF is obtained by one-dimensionally rearranging quantization values of 
DCT coefficients and encoding quantization values other than zero run-length and zero value. If all the quantization val- 
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ues are zero, no DCT-COEF is generated. 

[0047] The count value s of the object counter 41 is reset to zero upon start of input of 1 -frame code data. Then, the 
number of occurrence of SC indicative of the header of object of the code data is counted. The result of counting is 
inputted into the object combiner 43. The object decoder counter 42 counts the number d of object decoders. In the 
5 present embodiment, as the number of object decoders is three, accordingly, the output from the object decoder coun- 
ter 42 is "3". 

[0048] Fig. 10 is a block diagram showing the construction of the object combiner 43. A terminal 61 inputs the code 
data 8. A terminal 62 inputs the number s of the objects from the object counter 41. A terminal 63 inputs the number d 
of the object decoders from the object decoder counter 42. 

io [0049] A code memory 64 is used for storing code data for one or more frames inputted from the terminal 61 . A loca- 
tion information extractor 65 extracts the Loc code from the code data stored in the code memory 64. and stores the 
extracted Loc code into a location information memory 66 in frame units. An object distance calculator 67 calculates 
distances between respective objects, based on the Loc code stored in the position location memory 66. 
[0050] A distance comparator 68 compares the distances calculated by the object distance calculator 67, and selects 

is objects to be combined, based on the result of comparison. A selector 69 outputs object code data, read from the code 
memory 64, to a code divider 70 or terminal 76, designated by the distance comparator 68 as an output destination, for 
each object. 

[0051 ] The code divider 70 divides object code data into Loc. Size, Shape and Texture code. A location code combiner 
71 combines the Loc code of two objects into one Loc code. A size code combiner 72 combines the Size code of two 
so objects into one Size code. A shape code combiner 73 combines the Shape code of two objects into one Shape code. 
A texture code combiner 74 combines the Texture code of two objects into one Texture code. A code synthesizer 75 
synthesizes outputs from the location code combiner 71 to the texture code combiner 74 into one code data. 
[0052] One of the output from the code synthesizer 75 and that from the selector 69 is forwarded to the next stage via 
a terminal 76. 

25 

[Operation] 

[0053] Next, the operation of the present embodiment will be described on a case where an MPEG Intra frame or 
respective Motion JPEG frames are independently encoded. 

30 

• Frame- Based Coding 

[0054] In Fig. 10. code data for one or more frames is stored via the terminal 61 into the code memory 64. and the 
number s of objects is inputted via the terminal 62 from the object counter 41. In case of the code data in Fig. 8, the 
35 number s of objects is four (s=4). Further, the number d of object decoders is inputted via the terminal 63 from the object 
decoder counter 42. In the present embodiment, the number d of object decoders is three (d=3). Accordingly, s-d=1 
holds, i.e., the construction has one less decoder. 

[0055] The object distance calculator 67 obtains the distance between the object 21 and the object 22 from the Loc 
code stored in the location information memory 66. If the location of the object 21 is (xl .y1) and that of the object 22 is 
40 (x2,y2), the distance D12 between these objects is represented by the following equation: 

D12 = 7{(x1-x2) 2 + (y1-y2) 2 } (1) 

[0056] Similarly, the distance D1 3 between the abject 21 and the object 23 and the distance D23 between the object 
45 22 and the object 23 are obtained. Based on the obtained distances between objects, the distance comparator 68 
selects a plurality of objects with a short distance therebetween, to combine the objects for compensation of the short- 
age of object decoder. For example, the distance comparator 68 selects a plurality of objects with the smallestsum of 
distance therebetween in a plurality of frames. In the present embodiment as the shortage of object decoder is "1 ". two 
objects are combined into one object. If the sum of distance Dl 2 is the smallest, the object 21 and the object 22 are 
so combined. Thus, the shortage of object decoder is resolved. 

[0057] If the object 21 and the object 22 are combined, the selector 69. controlled by the output from the distance 
comparator 68. sends the header outputted from the code memory 64 to the terminal 76. and sends the Object 0 as 
background object to the terminal 76. 

[0058] Next, the output from the distance comparator 68 for the Object 1 corresponding to the object 21 indicates 
55 "selection", accordingly, the selector 69 sends the Object 1 to the code divider 70. The Loc code of the Object 1 is sent 
to the location code combiner 71 , the Loc code and Size code are sent to the size code combiner 72. the shape code 
.. . is sent to the shape code combiner 73. and the Texture code is sent to the texture code combiner 74. Next, the output 
from the distance comparator 68 for the Object 2 corresponding to the object 22 also indicates "selection", accordingly, 
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the Object 2 code is divided into the respective code, and the divided code are inputted into the location code combiner 
71 to the texture code combiner 74, as in the case of the Object 1 . 

[0059] Note that as the output from the distance comparator 68 for the Object 3 corresponding to the object 23 indi- 
cates non-selection", the code data of the object 23 is outputted to the terminal 76 without any processing Further for 
a frame having the code data 8 where s-d>0 holds, object combining is not performed, and the output from the selector 
69 is forwarded to the terminal 76. 

[0060] The location code combiner 71 decodes the respective Loc code, and obtains location information (x1 yl) to 
(xn.yn) of the plurality of objects. Then, as represented by the following equation, the location code combiner 71 selects 
the minimum value of x- and y-coordinates from these location information, and outputs new location information (xV 

(xV.yV) = (min(x1.x2 xn). min(y1,y2 yn)) ( 2 ) 

n: the number of combined objects 

[0061] The size code combiner 72 decodes the respective Loc and Size code, and obtains location information and 
size ,nformat.on (x1 ,y 1), (Sx1 ,Sy 1) to (xn.yn). (Sxn.Syn) of the plurality of objects. Then, the size code combiner 72 cal- 
culates new location mformation (x1\ y r) from the equation (2), and obtains new size information (Sx1\Sy1') from the 
following equation and outputs the information. 

(SxV.Syr) = (max(x1+Sx1,x2+Sx2,...,xn+Sxn)-x1\ max(y1 +Sy1 f y2+Sy2 yn+Syn)-yV) (3) 

[0062] The s hape code combiner 73 generates ^ synthesized from the shapes of the plurality of objects. When 
the objects 21 and 22 are synthesized, the shape of a new object is represented by a mask 80 as shown in Fig 1 1 A 
The original masks 24 and 25 remain the same, and the portion other than the masks are newly added In Fig 11 A the 
value of the hatched portion is the same as that of the solid black masks 24 and 25. Accordingly, as zero-run'has 
increased on the right side of the mask 24, zero run-length is added after code indicative of a change point nearest to 
the frame right end. 

[0063] Further, if another object does not exist on the right side of the object 21. the above-described change point 
merely indicates the final change point of the line, and the code does not increase. On the other hand if another object 
!u 6 r ' 9 ^ Side ° f the ° bjeCt 21 ' Z6ro run " ,en 9 th corresponding to the number of pixels between both objects is 
added to the code. That is. the code can be replaced with code to which zero run-length is added. Further if there is a 
third object on the right side of the other object on the right side of the object 21. the code of the object 21 is replaced 
with code where the zero run-length corresponding to the interval between the objects have been added the code of 
the object 2l^The replaced code is outputted as new Shape code. Note that with respect to a line including no object 
almost no code is generated. ^ ' 

[0064] The texture code combiner 74 generates code synthesized from textures of the plurality of objects Fiq 1 1 B 
shows a status where the texture of the object 21 and that of the object 22 are synthesized. The texture of a new object 
is represented as an object 81 . The original objects 21 and 22 remain the same, and a hatched portion other than the 
objects .s newly added. Note that the value of the hatched portion is zero. In the MPEG coding or the like the DC com- 
ponent of a pixel of interest is converted into a difference between the DC component and that of a left block Further 
quantization values of AC components are one-dimensionally arrayed, and zero run-length and nonzero values are 
encoded In the hatched portion in Fig. 1 1B, the difference between the DC component of a pixel of interest and that of 
a left block is zero, and the values of all the AC components are zero. In this case, in the MPEG1 coding, in macro-block 
units. 1 bit ind.cat.ve of macro block type, luminance 12 bits and chromaticity 4 bits indicative of DC component size 
and EOB (End of Block) 12 bits indicative of the end of the block, i.e.. total 29 bits, are added. In this manner. Texture 
code of the object 81 where the textures of the plurality of objects are combined is generated, and the Texture code is 
outputted. 

[0065] The code synthesizer 75 synthesizes outputs from the location code combiner 71 to the texture code combiner 
74. to generate code data of the combined object, and outputs the code data to the terminal 76. 
[0066] Fig. 9 shows an example of code data synthesized as above. The code data Object 1 of the object 21 and the 
code data Object 2 of the object 22 in Fig. 8 are combined into code data Object 1 \ Note that the code data Object 3 of 
the object 23 remains the same. 

[0067] The code data processed as above is inputted into the demultiplexer 9. and the object code data is divided into 
Object 0. Object r and Object 3. Then, the code data Object 0 is inputted into the object decoder 10; the code data 
Object r is inputted into the object decoder 1 1 ; and the code data Object 3 is inputted into the object decoder 12 Trie 
respective object decoders output location information obtained by decoding the code data and the image data to the 
object compositer 14. The object compositer 14 arranges the image data in accordance with the location information of 
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the respective objects, to obtain a reproduced image 15. 
-Moving Image Coding 

5 [0068] In moving image coding, the coding efficiency is improved by motion compensation. As an example, coding by 
using the correlation between frames such as a predicted frame in MPEG standard will be described. 
[00691 Fig. 12 is an-example of code data 8 for 1 frame to be motion-compensated. Similar to the code data in Fig. 8, 
the code data in Fig. 12 has a header, code data Object 0 indicative of background object, and code data (Object 1 to 
Object 3) of respective objects. Each code data comprises the SC incficative of the header of the object, the Loc code 

w indicative of location, the Size code indicative of size, the Shape code indicative of shape and Texture code indicative 
of texture. In the MEGP coding, an object is divided into macro blocks and motion compensation is performed in block 
units. As a result. The Texture code comprises MV code indicative of motion vector as a result of motion vector coding, 
and DCT-COEF quantized and encoded from the result of block-based division and DCT conversion. 
[0070] In moving image coding, the object number regulator 40 performs similar processing to that in frame-based 

is coding as described above. Although detailed explanation will be omitted, if the shortage of object decoder is "1 " and 
the distance D12 is the shortest, the objects 21 and 22 are combined, to reduce the number of objects. Further, the 
motion vectors included in the respective objects and the result of coding of DCT coefficients are not changed. 
[0071 ] In the MPEG coding or the like, when frame correlation is utilized, the DC component and AC components of 
predicted difference are encoded. Further, a motion vector is converted to the difference between the motion vector and 

20 that of a left macro block. Accordingly, in the hatched portion in Fig. 1 1 B, the difference of the motion vector is zero, and 
the values of all the AC components are zero. In the MPEG coding, such macro block is not encoded and is skipped, 
and corresponding code merely indicates the number of skipped macro blocks. Accordingly, in the object combining, 
the code of macro block included in an object which appears next is merely changed, and the code is merely slightly 
changed. In this manner, the Texture code of combined objects is generated and outputted. 

25 [0072] Fig. 13 is an example of code data synthesized as above. The code data Object 1 of the object 21 and the 
code data Object 2 of the object 22 in Fig. 12 are combined into code data Object 1\ Note that the code data Object 3 
of the object 23 remains the same. 

[0073] As described above, in the present embodiment, in a moving image encoded in object units, if the number of 
coded objects is greater than that of decoders, i.e.. there is shortage of decoders, to reduce the number of objects in 

30 correspondence with the shortage, a plurality of objects with a short distance therebetween are combined. This enables 
efficient and proper reproduction of moving image including a number of coded objects, with a limited number of decod- 
ers. Further, the objects are synthesized in code data status. That is, as the combining is made by change or addition 
of code, the objects can be synthesized at a high speed, and further, increment in code length is very small. 
[0074] Further, in the above description, a coded moving image is decoded, however, a coded still image can be sim- 

35 ilarly processed. That is, the above-described frame-based decoding can be applied to still image decoding. For exam- 
ple, as shown in Fig. 14A, an image 90 includes character areas 91 and 94 and photographic areas 92 and 93. and the 
characters are encoded by the MMR (Modified Modified Read) coding and the photographs are encoded by the JPEG 
coding. If only one decoder for the MMR coding and only one decoder for the JPEG coding are prepared, the image 90 
can be decoded by combining and dividing the respective areas into objects 95 and 96 as shown in Figs. 14B and 14C. 

ao [0075] Note that in the above description, the Shape code is encoded by the MMR coding, and the Texture code is 
encoded by the MPEG coding, however, the present invention is not limited to these coding methods. Further, the func- 
tion of the demultiplexer 9 may be incorporated into the object number regulator 40. Further, if s-da2 holds as the dif- 
ference between the number d of object decoders and the number s of objects included in code data, objects with a 
short distance therebetween, i.e., 2 • (s-d) objects may be combined with (s-d) objects, or (s-oV1 ) objects with the 

45 shortest distance therebetween may be combined into one object. 

[0076] In the above description, code data of regulated number of objects are inputted into the object decoders. How- 
ever, if it is arranged such that the code data of regulated number of objects are temporarily stored in the storage device 
50 as shown in Fig. 7, and the code data of regulated number of objects are read from the storage device 50 and 
decoded, decoding processing can be performed at a speed higher than that in decoding with object combining. 

so [0077] Further, according to the present embodiment, the number of object decoders is not limited. Accordingly, the 
processing capability can be easily improved by increasing the object decoders. Further, the location of an object may 
be obtained precisely by decoding, e.g.. the Shape code, as well as utilizing the location code. 

Second Embodiment 

55 . 

[Construction] 

[0078] Fig. 1 6 is a block diagram showing the construction of the object combiner 43 according to a second embodi- 
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ment of the present invention. In Fig. 16, elements corresponding to those in the construction of Fig. 10 have the same 
reference numerals, and detailed explanations of the elements will be omitted. 

[0079] Numerals 1 1 1 and 112 denote code memories having a similar function to that of the code memory 64; 1 13 
and 114, location information memories having a similar function to that of the location information memory 66; and 

s 1 15, an object motion calculator which detects the movement of object. 

[0080] Numeral 1 1 6 denotes a determination unit 1 1 6 which determines whether or not object combining is necessary 
and determines objects to be combined, based on the results of calculation by the object motion calculator 1 15 and the 
object distance calculator 67, and the number s of objects and the number d of object decoders inputted from the ter- 
minals 62 and 63. Numeral 1 17 denotes a selector which outputs object code data read from the code memory 1 1 1 , or 

10 from the code memory 1 1 2, if necessary, to an output destination designated by the determination unit 1 16. 

[Operation] 

[0081 ] Next, the operation of the object combiner 43 as shown in Fig. 16 will be described. First, frame-based coding 
is to independently encode respective frames such as Intraframe coding in the MPEG standard or Motion JPEG coding 
will be described. 

[0082] In this case, the code memory 1 1 1 has the same function as that of the code memory 64 in Fig. 10; the location 
information memory 1 13 has the same function as that of the location information memory 66 in Fig. 10; and the deter- 
mination unit 1 16 has the same function as that of the distance comparator 68 in Fig. 10. Accordingly, the code data for 

20 1 frame inputted from the terminal 61 is stored into the code memory 111. The number s of objects, as the output from 
the object counter 41, is inputted into the terminal 62. The number d of object decoders, as the output from the object 
decoder counter 42, is inputted into the terminal 63. The location information extractor 65 extracts location Information 
of respective objects from the code data stored in the code memory 111, and inputs the extracted information into the 
location information memory 113. The object distance calculator 67 calculates distances between objects based on the 

25 location information stored in the location information memory 1 13. The determination unit 1 16 determines whether or 
not object combining is necessary from the number s of objects and from the number d of object decoders. If object 
combining is necessary, the determination unit 1 16 determines the number of objects to be combined, then compares 
the distances between objects obtained by the object distance calculator 67. and determines the necessary number of 
objects to be combined. 

so [0083] The object code data read from the code memory 1 1 1 is inputted into the selector 1 1 7. The selector 1 1 7 for- 
wards code data of header, background object and uncombined objects to the terminal 76. On the other hand, the 
selector 1 1 7 inputs code data of objects to be combined to the code divider 70. The code data inputted into the code 
divider 70 is divided into location, size, shape and texture code data, and inputted into the location code combiner 71, 
the size code combiner 72. the shape code combiner 73 and the texture code combiner 74, respectively. Object code 

35 data, combined in a procedure similar to that described in the first embodiment, is outputted from the terminal 76. 
[0084] Next, a frame encoded by using the correlation between frames such as a predicted frame in the MPEG cod- 
ing, will be described. In this case, the MPEG-coded frame 20 in Fig. 1 and a frame 100 shown in Fig. 15, following the 
frame 20, will be described. Note that in the frame 100, the object 21 (helicopter) has moved rightward, and the object 
22 (train) and the object 23 (car) have moved leftward, with respect to the frame 20. 

40 [0085] Prior to processing, the code memories 1 1 1 and 1 1 2 and the location information memories 1 1 3 and 1 14 are 
cleared, and the other elements are initialized. The number s of objects is inputted into the terminal 62, and the number 
d of object decoders is inputted into the terminal 63. First, the code data of the frame 20 is inputted into the terminal 61 , 
and stored into the code memory 111. The location information extractor 65 stores location information of the respective 
objects in the frame 20 into the location information memory 113. The object distance calculator 67 obtains distances 

45 between the respective objects in the frame 20. 

[0086] Next, the code data in the code memory 1 1 1 is moved to the code memory 112, and the location information 
in the location information memory 113 is also moved to the location information memory 114, then the code data of 
the frame 100 is inputted into the terminal 61 and stored into the code memory 111. The location information extractor 
65 stores location information of the respective objects in the frame 100 into the location information memory 113. 

so [0087] The object motion calculator 1 1 5 calculates the motions of the respective objects from the locations of the 
respective objects in the location information memories 1 13 and 1 1 4. In case of the object 21 , assuming that its location 
in the frame 20 is (x21 1.y21 1) and that in the frame 100 is (x212,y212), the motion vector MV21 = (mv21x,mv21y) is 
represented as: 

55 MV21 = <mv21x,mv21y) = ((x212-x211).(y212-y211)) (4) 

[0088] Regarding the objects 22 and 23, motion vectors MV22 and MV23 are obtained in a similar manner. 

[0089] The distances D1 2, D 13 and D23 as the outputs from the object distance calculator 67, and the motion vectors 
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MV21, MV22 and MV23 as outputs from the object motion calculator 1 15 are inputted into the determination unit 116. 
The determination unit 1 1 6 determines whether or not object combining is necessary, from the number s of objects and 
the number d of object decoders, and if the object combining is necessary, determines the number of objects to be com- 
bined and objects to be combined. 
5 [0090] In this case, objects having motion vectors with directions close to each other are determined as objets to be 
combined. Then, the difference vectors between the motion vectors of the respective objects are obtained, and motion 
vector(s) less than a threshold value Thdv is selected. That is. the difference vector DV2122 between the motion vector 
MV21 of the object 21 and the motion vector MV22 of the object 22 is represented by the following equation: 

to DV2122 = (dv2122x.dv2122y) = ((rnv21x-mv22x),(mv21y-mv22y)) (5) 

[0091 ] The size D2122 of the difference vector DV2122 is represented by the following equation: 

D2122 = 7(dv2122x 2 +dv2122y 2 ) (6) 

15 

[0092] All the difference vector sizes are obtained. The obtained difference vector sizes D2122. D2223 and D2123 
are compared with the threshold value Thdv, and the difference vector size(s) less than the threshold value is selected. 
As the objects 22 and 23 have moved in the same direction, the difference vector size D2223 of the difference vector is 
less than that with respect to the object 21 . If only the difference vector size D2223 is less than the threshold value Thdv, 

so the objects to be combined are the objects 22 and 23. If all the difference vector sizes are less than the threshold value 
Thdv. objects with the shortest distance therebetween are selected as objects to be combined. Further, if there is no 
difference vector size less than the threshold value, objects with the shortest difference therebetween are combined. 
[0093] Then, object combining is performed based on the determination. In this case, the object 22 and the object 23 
are combined so as to reduce the number of objects. This operation will be described using the code data 8 in Fig. 8 as 

25 an example. 

[0094] First, the selector 1 17 reads the header from the code memory 1 12 and outputs the header via the terminal 
76. Further, the selector 1 17 reads the code data Object 0 of the background object, and outputs the code data via the 
terminal 76. As the object 21 is not combined, the selector 1 17 similarly reads the code data Object 1 and outputs the 
code data via the terminal 76. 

30 [0095] Then, as the next code data Object 2 corresponds to the object 22. the selector 1 1 7 reads the code data Object 
2 from the code memory 1 12 and inputs the code data into the code divider 70. The code divider 70 inputs the Loc code 
from the object code data into the location code combiner 71 . the Loc code and the Size code from the object code data, 
into the size code combiner 72, the Shape code from the object code data, into the shape code combiner 73. and the 
Texture code from the object code data, into the texture code combiner 74. 

35 [0096] Next, the selector 1 1 7 reads the code data Object 3 of the object 23 to be combined with the object 22. from 
the code memory 112, and inputs the code data into the code divider 70. As in the case of the code data Object 2, 
divided code are respectively oulputted. 

[0097] The location code combiner 71 decodes the respective Loc code, and generates new location information 
(x2\y2') from the location information (x2,y2) and (x3.y3).of the two objects, based on the equation (2), then encodes 
40 the new location information (x2\y2'), and outputs the coded location information, 

[0098] The size code combiner 72 decodes the Loc code and the Size code, then generates new size information 
(Sx2\Sy2-) from the location information (x2.y2) and (x3.y3) and size information (Sx2,Sy2) and (Sx3.Sy3) of the two 
objects, based on the equation (3), then encodes the new size information (Sx2',Sy2') ( and outputs the coded size infor- 
mation. 

45 [0099] The shape code combiner 73 generates code of a shape synthesized from the shapes of the two objects. 
When the objects 22 and 23 are synthesized, the shape of the new object is represented by a mask 150 in Fig. 17A. 
That is, in Fig. 1 7A, a hatched portion is newly added to a mask 25 of the object 22 and a mask 26 of the object 23, as 
the mask 150. Note that the value of the hatched portion is the same as that of the mask 80 in Fig. 1 1 A Then, as in the 
case of the first embodiment, addition of zero-run and/or code change is performed, and the obtained code is outputted 

so as new Shape code. 

[0100] The texture code combiner 74 generates code of texture synthesized from the textures of the two objects. Fig. 
1 7B shows a status where the texture of the object 22 and that of the object 23 are synthesized. That is. a texture having 
zero value, as represented as a hatched portion, is added to the textures of the objects 22 and 23. Then, as in the case 
of the first embodiment, code is added in macro block units in the hatched portion or the number of skipped blocks is 
55 changed, thus Texture code of an object 1 51 is generated and outputted. 

[01 01 ] The code synthesizer 75 synthesizes the outputs from the location code combiner 71 to the texture code com- 
biner 74. to generate code data of the combined object. The code data of the combined object is outputted from the 
terminal 76. 
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[0102] Fig. 18 is an example of code data including the combined object. In Fig. 18, the code data Object 1 of the 
object 21 remains the same, while the code data Object 2 of the object 22 and the code data Object 3 of the object 23 
are combined as code data Object 2'. 

[0103] The generated code data is inputted into the demultiplexer 9, and divided into the code data Object 0, the 
5 Object 1, and the Object 2'. The code data object 0 is inputted into the object decoder 10; the code data Object 1 is 
inputted into the object decoder 11 ; and the code data Object 2' is inputted into the object decoder 12. The object 
decoders 10 to 12 decode the code data, generate location information and image data of the respective objects, and 
output them to the object compositer 14. The object compositer 14 arranges the image data in accordance with the 
location information of the respective objects, thus obtains the reproduced image 15. 
10 [0104] In the present embodiment, in a moving image encoded in object units, if the number of coded objects is 
greater than that of decoders, objects are combined, from objects with motion vectors or moving amounts close to each 
other, whereby original image reproduction can be efficiently made even by a limited number of decoders. Further, as 
change or addition of code is performed in the form of code data, the processing can be made at a high speed, and 
increment in code length is very small. Further, as the decoding load on the respective decoders can be uniformed. Fur- 
15 ther, even if the currently -processed frame is not a frame encoded by the Intraframe coding, upon occurrence of scene 
change, objects to be combined in interframe coding can be re-determined. 

[0105] In the present embodiment, the difference vector size and the distance between objects are used for determi- 
nation of objects to be combined, however, the determination may be made by only using the difference vector size. Fur- 
ther, in the present embodiment, the Shape code is encoded by the MMR coding, and the Texture code is encoded by 

20 the MPEG coding, however, the present invention is not limited to these coding methods. 

[0106] Further, in the present embodiment, the function of the demultiplexer 9 may be incorporated into the object 
combiner 40. Further, the number of object decoders and the number of objects included in code data are not limited 
to those in the embodiment. As long as (s-d)^2 holds, 2 • (s-d) objects can be combined to (s-d) objects, from objects 
with the minimum difference vector size, or (s-d+t ) objects can be combined into one object from objects with the min- 

25 imum difference vector size, or combination between the former and latter cases may be employed. 

[0107] In the present embodiment, the decoding apparatus having decoders to outputs decoded results has been 
described, however, if it is arranged such that code outputted from the object combiner 43 is temporarily stored into the 
storage device 50, and the code read out of the storage device 50 is decoded, object combining is unnecessary, and 
high-speed decoding (image reproduction) is possible. 

30 [01 08] Further, in the present embodiment, as the number of object decoders can be freely set, the number of object 
decoders can be easily increased so as to improve processing capability. Further, the motion calculation may be made 
by referring to the motion vectors of objects as well as referring to the location information of the objects. 

Third Embodiment 

35 

[0109] Fig. 1 9 is a block diagram showing the construction of the object combiner 43 according to a third embodiment 
of the present invention. In Fig. 1 9. elements corresponding to those in Fig. 10 have the same reference numerals, and 
detailed explanations of the elements will be omitted. 

[01 10] A code length extractor 200 extracts code lengths of respective objects of code data stored in the code memory 
40 64, and stores the extracted code lengths into a code length memory 201 . A code length comparator 202 compares the 
respective code lengths of the objects, stored in the code length memory 201 , with each other, then determines whether 
or not object combining is necessary, and determines objects to be combined. 

[0111] If object combining is performed, objects to be combined are determined, sequentially from objects with short 
code lengths. For example, if the number s of objects is four (s=4), and the number d of object decoder is three (d=3), 

45 s-d=1 holds, accordingly, two objects having short code lengths are combined into one object. If the code data of the 
frame 20 in Fig. 1 is as shown in Fig. 20. the code data Object 2 of the object 22 is the minimum code data, and the 
code data Object 1 of the object 21 is the next minimum code data. In this case, the code data Object 1 and Object 2 
are combined. The operations of other elements are the same as those in the above embodiments. From the terminal 
76, code data as shown in Fig. 21 is outputted. 

so [01 1 2] Then, the code data Object 1 and Object 2 as the code data of the object 2 1 and the object 22 are combined 
in all the frames. The details of the combining are as described in the above respective embodiments. Even a motion- 
compensation frame or a frame encoded by the Intraframe-coding of the MPEG coding or the like are included in the 
image data, the object combining is made in a similar manner to that in the above respective embodiments. 
[01 1 3] According to the present embodiment, simitar advantages to those in the above respective embodiments can 

55 be obtained. Further, in case of still image as shown in Fig. 14A, a character image is encoded at a high compression 
rate by the MMR coding, and the resulting code length is short. Accordingly, if an image where character portions are 
. combined as shown in Fig. 14B is handled as one object, similar advantages to those as above can be obtained in the 
still image in Fig. 14 A. . 
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Fourth Embodiment 

[0114] In the MPEG coding or the like, a frame within which coding is performed (a frame encoded by the Intraframe- 
coding) and a frame encoded by using irrterframe correlation (a frame encoded by the Interframe-coding) are treated. 
£ The frame encoded by the Intraframe-coding is inserted to ensure synchronism or to prevent accumulation of DCT dif- 
ferences. 

[01 1 5] The present embodiment re-determines objects to be combined upon coding of Intraframe-coding frame. 
[01 1 6] Fig. 22 is a block diagram showing the construction of the object combiner 43 according to a fourth embodi- 
ment of the present invention. In Fig. 22, elements corresponding to those in Fig. 1 0 have the same reference numerals, 
io and detailed explanations of the elements will be omitted. 

[01 17] NumeraJ 301 denotes a header analyzer which analyzes the header of each frame. Numeral 302 denotes a 
distance comparator having approximately the same operation as that of the distance comparator 68 in Fig. 10. As in 
the case of the first embodiment, prior to processing, the number s of objects and the number d of object decoders are 
inputted. If s<d holds, code inputted into the terminal 61 is outputted from the terminal 76 without any processing. 
is [01 18] On the other hand, if s>d holds, the header of code data for 1 frame, inputted from the terminal 61 and stored 
into the code memory 64. is inputted into the header analyzer 301. In the header with description of frame attribute, 
information indicating whether or not the frame is a frame encoded by the Interframe-coding. i.e., a frame to be-encoded 
by using irrterframe correlation, is described. For example, the MPEG coding handles an I frame which is encoded 
within the frame without interframe correlation (by Intra coding), and a P and B frames encoded by using interframe cor- 
se relation with motion-compensation. 

[01 1 9] When a frame encoded without interframe correlation is detected from the result of header analysis, the oper- 
ation of the present embodiment is as follows. Code data is read out of the code memory 64. The location information 
extractor 65 extracts the Loc code following the SC of respective objects, and stores the extracted Loc code into the 
location information memory 66. The object distance calculator 67 obtains distances between the objects, and the dis- 
ss tance comparator 302 selects, sequentially from objects with the shortest distance therebetween. Note that the proce- 
dure of selection is similar to that of the first embodiment. Information indicative of the selected objects are held in the 
distance comparator 302. 

[01 20) The information indicative of the selected objects held in the distance comparator 302 is updated only if a new 
instruction is inputted from the header analyzer 301, i.e.. only if a frame encoded without interframe correlation has 
30 been newly detected. 

[0121] On the other hand, in a frame encoded by using interframe correlation, object combining is performed in 
accordance with information indicative of selected objects held in the distance comparator 302. and code of a new 
object obtained by combining objects is outputted from the terminal 76, as in the case of the first embodiment. 
[01 22] In this manner, according to the present embodiment, objects to be combined are re-determined upon decod- 
es ing of frame encoded by the Intraframe-coding, whereby change of coding efficiency by object combining can be sup- 
pressed. Even if a frame encoded by the Intraframe-coding is not detected, when scene change occurs, objects to be 
combined are re-determined, even with objects encoded by using interframe correlation. Regarding scene change, in 
a P frame, for example, if the number of macro blocks to be Intra-encoded is large, or in a B frame, if a frame where 
motion vectors are referred to greatly depends on its pervious or subsequent frame, it is determined that scene change 
40 has occurred. 

[01 23] According to the fourth embodiment, as in the case of the first embodiment, by re-determining objects to be 
combined in a frame encoded by the Intraframe-coding, change of coding efficiency due to object combining can be 
suppressed. 

45 Modifications of First to Fourth Embodiments 

[0124] As shown in Fig. 23, the header analyzer as described in the fourth embodiment can be added, as a header 
analyzer 401 . to the construction in Fig. 1 6 of the second embodiment. That is. as a result of frame header analysis, if 
it is determined that the frame has been encoded without interframe correlation, a determination unit 402 determines 
so objects to be combined, based on distances between objects outputted from the object distance calculator 67 and 
motions of objects outputted from the object motion calculator 1 15. as in the case of the second embodiment. Informa- 
tion indicative of objects to be combined is held in the determination unit 402. and only H an instruction is inputted from 
the header analyzer 401. the held content is updated. 

[0125] As shown in Fig. 24. the header analyzer as described in the fourth embodiment can be added, as a header 
55 analysis 501 . to the construction in Fig. 1 9 of the third embodiment That is. as a result of frame header analysis, if it is 
determined that the frame has been encoded without interframe correlation, a code length comparator 502 determines 
objects to be combined, based on code lengths of respective objects, as in the case of the third embodiment. Informa- 
tion indicative of objects to be combined is held in the code length comparator 502. and only if an instruction is inputted 
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from the header analyzer 501, the held content is updated. 

[0126] According to the constructions in Figs. 23 and 24, objects to be combined are re-determined upon decoding 
of frame encoded by the Intrafrarne-coding, whereby change of coding efficiency due to object combining can be sup- 
pressed. 

5 [01 27] Further, in the MPEG4 standard, handling of sound data as an object is studied. If a distance between sound 
sources of sound objects is regarded as a distance between objects, the first embodiment is applicable, and if the 
movement of sound source is object motion, the second embodiment is applicable. In use of code lengths of respective 
objects, the third embodiment is applicable. Thus, the above-described respective embodiments are applicable to cod- 
ing of sound including audio information. 

10 [01 28] As described above, the first to fourth embodiments provide decoding apparatus and method which decode 
all the objects even if the number of decoders is limited. 

[01 29] Further, the embodiments provide decoding apparatus and method which decode a coded still image without 
degrading the image quality even if the number of decoders is limited. 

[01 30] Further, the embodiments provide decoding apparatus and method which decode a coded moving image with- 
75 out degrading the image quality even if the number of decoders is limited. 

Fifth Embodiment 

[Construction] 

20 

[0131] Fig. 29 is a block diagram showing the construction of a moving image processing apparatus according to a 
fifth embodiment of the present invention. In the present embodiment, the MPEG4 coding is used as a moving image 
coding method. Note that the coding method of the present embodiment is not limited to the MPEG4 coding, but any 
other coding method can be employed as long as it respectively encodes a plurality of objects within an image. 

25 [0132] In Fig. 29, numeral 1201 denotes an encoder which inputs a moving image and encodes the image by the 
MPEG4 coding of Core profile and level 2. Numeral 1202 denotes a storage device used for storing coded moving 
image data. The storage device 1202 comprises a magnetic disk, an magneto-optic disk or the like. As the storage 
device 1202 is removably attached to the moving image processing apparatus, coded moving image data can be read 
in another apparatus. Numeral 1203 denotes a transmitter which transmits encoded moving image data to a LAN or a 

so communication line, and performs broadcasting or the like; 1204, a receiver which receives code data outputted from 
the transmitter 1203; 1 205, a prof ile and level regulator to which the present invention is applied; 1206. a storage device 
used for storing output from the profile and level regulator 1 205; 1 207. a decoder which decodes code data encoded by 
the MPEG4 coding of Core profile and level 1 ; and 1208, a display unit which displays a moving image decoded by the 
decoder 1207. Note that as described above, the encoder 1201 performs coding of Core profile and level 2, and in this 

35 example, to simplify the explanation, the encoder 1201 performs coding at a bit rate of 384 kbps. 

[0133] Fig. 43 shows an example of an image to be encoded. In Fig. 43, respective numerals denote objects. An 
object 2000 represents background; an object 2001, a balloon moving in the air; an object 2002, a bird; objects 2003 
and 2004. a woman and a man. 

[01 34] Fig. 31 A shows a bit stream when the image in Fig. 43 is encoded. The bit stream has an arrangement infor- 
40 mation a indicative of location information of objects 2000 to 2004 at its head. Actually, the arrangement information a 
is encoded in BIFS (Binary Format for Scene description) language to describe scene construction information, and the 
arrangement information a is multiplexed. Then, VOSSC. Visual Object data a-1 , a-2. a-3 and VOSEC follow. The code 
data in Rg. 31 A is stored into the storage device 1202 or transmitted via the transmitter 1203. The code data is inputted 
via the storage device 1202 or the receiver 1204, into the profile and level regulator 1205 as a characteristic element of 
45 the present invention. The profile and level regulator 1 205 also inputs the status of the decoder 1 207. 

[01 35] Fig. 30 is a block diagram showing the detailed construction of the profile and level regulator 1205. In Fig. 30, 
numeral 1101 denotes the code data shown in Fig. 31 A; 1 102, a separator which separates the code data 1 101 into 
code data indicative of arrangement information and header information, and code data indicative of respective objects; 
1 103, a header memory for storing code data indicative of separated arrangement information and header information; 
so 1 104 to 1 108, code memories for storing code data for respective objects; 1 109, a profile and level extractor which 
extracts the PLI code from the code data 1 101 . and extracts information on the profile and level; and 1 1 10, an object 
counter which counts the number of objects included in the code data 1 101 . 

[01 36] Numeral 1111 denotes a decoder status receiver which obtains coding specifications (profile and level) of the 
decoder 1207 and other conditions; and 1 1 12. a profile and level input unit through which arbitrary profile and level are 
55 set from a terminal (not shown) or the like; 1 1 13. a profile and level determination unit which compares outputs from the 
profile and level extractor 1 109 and the object counter 1110 with profile and level information inputted from the decoder 
. status receiver 1 1 1 1 or the profile and level input unit 1112, and determines whether or not the number of objects must 
be regulated. 
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[0137] Numeral 1 1 14 denotes a code length comparator which determines the order of code lengths of objects by 
counting the code lengths of objects when the code data 1101 is inputted and comparing the code lengths with each 
other; 1 1 15, a header changer which changes the content of header information stored in the header memory 1103, 
based on the outputs from profile and level determination unit 1 1 13 and the code length comparator 1 1 1 4; 1 1 1 6. a mul- 
£ tiplexer which multiplexes code data read from the code memories 1 104 to 1 108 based on the output from the header 
changer 1115 and the results of comparison by the code length comparator 1114; and 1117. code data outputted as a 
result of profile and level regulation. 

[Regulation of Profile and Level] 

10 

[0138] Hereinbelow, the processing in the profile and level regulator 1205 having the above construction will be 
described in detail. 

[0139] The code data 1 101 is inputted into the separator 1 102. the profile and level extractor 1 109. the object counter 
1110 and the code length comparator 1114. The separator 1 102 separates the code data 1 1 01 into code data indicative 

is of arrangement information and header information, and code data indicative of respective objects, and stores the 
respective code data into the header memory 1103 and the code memories 1104 to 1108. For example, the object 
arrangement information a. VOSSC, Visual Object SC. the respective code immediately prior to the VO data A. and the 
header information of VOL and VOP data in Fig. 25. and the like, are stored in the header memory 1 103. Further, the 
VOL and VOP data for the respective object, where the header information is removed, are stored in the code memories 

so 1 104 to 1 1 08. These data are stored independently such that the header-removed part is clearly indicated. For exam- 
ple, in the image in Fig. 43. as the number of objects is five, the code data of the objects 2000 to 2004 (VO data A to E 
in Fig. 31 A) are respectively stored into the code memories 1 104 to 1 108. 

[0140] At the same time, the object counter 1110 counts the number of objects included in the code data 1 101. Then 
the code length comparator 1114 measures code lengths of the respective objects. 
25 (0141 ] The profile and level extractor 1 1 09 extracts PLI-a from the code data 1 1 01 and decodes it, to extract informa- 
tion on the profile and level of the code data 1 101. At the same time of extraction, the decoder status receiver 1111 
operates, to obtain information on the profile, level and the like, decodable by the decoder 1 207. These information may 
be set by the user via the profile and level input unit 1112. 

[0142] The profile and level determination unit 11 13 compares the profile and level information, obtained from the 
30 decoder 1207. or set by the user, with the result of extraction by the profile and level extractor 1 109. If the obtained or 
set profile and level are higher than or equal to those extracted from the code data 1 101 , the profile and level determi- 
nation unit 1113 does not operate the header changer 1115. Then, the contents of the header memory 1 103 and the 
code memories 1 104 to 1 108 are read in the order of input, and multiplexed by the multiplexer 1116. Thus, code data 
1 1 1 7 is generated. That is. the contents of the code data 1 1 1 7 are the same as that of the code data 1 101 . 
35 [0143] On the other hand, if the profile and level obtained from the decoder 1207 or set by the user are lower than the 
profile and level extracted from the code data 1 101 . the profile and level determination unit 1 1 13 inputs the number of 
objects included in the code data 1101 from the object counter 1 1 10, and compares the number of objects with the 
number of decodable objects, determined from the obtained or set profile and level information. 
[0144] If the number of objects obtained by the object counter 1 110 is less than the number of decodable objects, the 
40 code data 1 1 1 7 is generated, as in the case of the above-described case where the obtained or set profile and level are 
higher than or equal to those extracted from the code data 1 101. 

[01 45] On the other hand, if the number of objects obtained by the object counter 1 1 1 0 is greater than the number of 
decodable objects, the number of decodable objects is inputted into the code length comparator 1 114. and the code 
lengths are compared with each other. The code length comparator 1114 sets objects to be decoded, from an object 

45 having the longest code length. That is. the objects are decoded sequentially from the object having the longest code 
length. For example, in Fig. 31 A. if the code length of video object becomes shorter, in the order in which the VO data 
A, the VO data D, the VO data C, the VO data E, and the VO data B appear, as the decoder 1207 performs decoding of 
Core profile and levei 1 , it can decode to a maximum of four objects. Accordingly, the code length comparator 1114 dis- 
ables reading of the VOdata B from the code memory 1 106, and enables reading from the code memories *1 104, 1 105, 

so 1107 and 1108. 

[0146] Then the profile and level determination unit 1113 operates the header changer 11 15 to change the content 
of PLI in correspondence with the decoder 1207. then, coding is performed. In this manner, header information on 
undecodable (deleted) object (VO data B in this case) by the decoder 1207 is deleted, based on the result of compari- 
son by the code length comparator 1114. That is. the header information of the code data 1 101 is rewritten with con- 
55 tents corresponding to the decoding capability of the decoder 1207 or the set profile and level. Further, arrangement 
information on the object 2002 corresponding to the deleted object (VO data B) is deleted from the arrangement infor- 
mation a, and new arrangement information (J is generated. 

[01 47] Then, the contents of the header changer 1 1 1 5 and the code memories 1 1 04. 1 1 05. 1 1 07 and 1 1 08 are read 
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in the order of input, and multiplexed by the multiplexer 1116, thus the code data 11 1 7 is generated. Fig 31 B shows a 
bit stream of the code data 1 1 1 7. In Fig. 31 B, the newly generated arrangement information p is provided at the head 
of the bit stream, then, VOSSC, Visual Object data p-1, p-2, p-3, and VOSEC follow. The Visual Object data p-i p-2 
and p-3 are obtained by regulating the number of objects with respect to the original Visual Object data a-1 a-2 and ct- 
3 in Fig. 31 A. For example, the Visual Object data p-1 comprises the Visual Object SC positioned at the head PLI-p 
tndicative of the profile and level corresponding to the decoder 1207, and code data where the code data <VO data B) 
on the object 2002 is deleted. 

[01 48] The code data 1117 obtained as above is stored into the storage device 1 206, or decoded by the decoder 1 207 
and displayed on the display unit 1 208. Fig. 44 shows a displayed image, represented by the decoded code data 1117 
In Fig. 44, the object 2002. representing the bird in the image as the object of encoding in Fig. 43, is deleted 
[01 49] Note that in the above description, the code length comparator 1 1 1 4 directly counts the code lengths from the 
code data 1101, however, the code length comparator 1 1 1 4 may count the code lengths based on the code data stored 
in the code memories 1 104 to 1 108. 

[0150] As described above, according to the present embodiment, even if the coding specifications (profile and/or 
level) of a decoder are different from those of an encoder, code data can be decoded. Further, by deleting object data 
having the shortest code length, selection of object to be deleted is facilitated, and the influence on a decoded image 
can be suppressed as much as possible. 

[0151] Further, even if the number of objects decodable by the decoder 1207 is less than the number defined by the 
coding specifications of the code data 1 101 . as the decoder status receiver 1111 obtains the number of actually deco- 
de dable objects, similar advantages can be attained. 

[0152] In addition, even when code data having coding specifications higher than or equal to those of the decoder 
1207 is inputted, by deleting object(s) to reduce the bit rate, decoding by the decoder 1207 can be performed. 
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Sixth Embodiment 



[0153] Heretnbetow, a sixth embodiment of the present invention will be described. Note that the general construction 
of the moving image processing apparatus according to the sixth embodiment is similar to that in Fig. 29 of the above- 
described fifth embodiment, therefore, an explanation of the construction will be omitted. 

[01 54] Fig. 32 is a block diagram showing the construction of the profile and level regulator 1 205 according to the sixth 
so embodiment of the present invention. In Fig. 32, elements corresponding to those in Fig. 30 have the same reference 
numerals and explanations of the elements will be omitted. In the sixth embodiment, the MPEG4 coding is employed 
as a moving image coding method, however, any other coding method is applicable as long as it encodes a plurality of 
objects within an image 

[01 55] In Fig. 32. numeral 1118 denotes a size comparator which extracts sizes of respective objects from the header 

35 memory 1 1 03 and compares the sizes with each other. 

[0156] As in the case of the fifth embodiment, the code data 1 101 is inputted into the separator 1102, the profile and 
level extractor 1 109. the object counter 1 1 10 and the code length comparator 1114, and the respective code data are 
stored into the header memory 1 1 03 and the code memories 1 1 04 to 1 1 08. At the same time, the object counter 1110 
counts the number of objects included in the code data. 

40 [01 57] The size comparator 1118 extracts an image size of each object, by extracting the respective VOL.width and 
VOL_height code in the bit stream structure in Fig. 25 and decoding the extracted codes. 

[0158] Then, as in the case of the fifth embodiment, the profile and level extractor 1 109 extracts information on the 
profile and level from the code data 1 101, and at the same time, information on profile and level and the like of the 
decoder 1207 is obtained from the decoder status receiver 1 1 1 1, or the profile and level are set by the user from the 

45 profile and level input unit 1112. 

[0159] The profile and level determination unit 1113 compares the profile and level information obtained from the 
decoder 1 207 or set by the user, as described above, with the result of extraction by the profile and level extractor 1 109. 
If the obtained or set profile and level are higher than or equal to the profile and level extracted from the code data 1101. 
profile and level determination unit 1113 does not operate the header changer 1115. Then, the code data 1117 similar 

so to the code data 1 1 01 is generated. 

[0160] On the other hand, if the profile and level obtained from the decoder 207 or set by the user are lower than the 
profile and level extracted from the code data 1101, the profile and level determination unit 1 1 13 inputs the number of 
objects included in the code data 1 101 from the object counter 1 1 10, and compares the input number with the number 
of decodable objects determined from the obtained or set profile and level. 

55 [0161] Then, if the number of objects obtained by the object counter 1 1 10 is less than the number of decodable 
objects, the code data 1 1 1 7 is generated as in the above-described case where the obtained or set profile and level are 
higher than or equal to those of the code data 1 101 . 

[0162] On the other hand, if the number of objects obtained by the object counter 1 1 10 is greater than the number of 
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decodable objects, the number of decodable objects is inputted into the size comparator 1118, and size comparison is 
performed. The size comparator 1118 sets a plurality of objects of the code data 1101, sequentially from the largest 
image size, as objects to be decoded. That is, the objects are decodable. sequentially from the largest image size. For 
example, in Fig. 43. in the image sizes of the respective objects, the image size becomes smaller in the order in which 
the objects 2000. 2004. 2001, 2003 and 2002 appear. As the decoder 1207 performs decoding Core profile and level 
1 . it can decode to a maximum of four objects. Accordingly, in the image in Fig. 43. except the smallest object 2002. the 
other four objects can be decoded. The size comparator 1118 disables reading of the code data of the object 2002 from 
the code memory 1 106, and enables reading from the code memories 1 104. 1 105. 1 107 and 1 108. 
[0163] Then, as in the case of the fifth embodiment, the profile and level determination unit 1 1 1 3 operates the header 
changer 1 115 to change the content of PLI in correspondence with the decoder 1207. then, coding is performed. Fur- 
ther, header information on the undecodable (deleted) object (object 2002 in this case) by the decoder 1207 is deleted, 
based on the result of comparison by the size comparator 1118. Further, arrangement information on the deleted object 
2002 is deleted from the arrangement information a. and new arrangement information p is generated. 
[0164] Then, the contents of the header changer 1115 and the code memories 1 104. 1 105. 1 107 and 1 108 are read 
in the order of input, and multiplexed by the multiplexer 1116. thus the code data 11 17 is generated. Fig. 31 B shows a 
bit stream of the code data 1 1 1 7 at this time. 

[0165] The code data 1117 obtained as above is stored into the storage device 1 206. or decoded by the decoder 1 207 
and displayed, as an image as shown in Fig. 44, on the display unit 1208. 

[0166] Note that in the above description, the size comparator 1118 extracts image sizes of objects based on the 
VOL_width and VOL_height code of the code data 1101. however, the size comparator 1118 may extract the image 
sizes based on the VOP_width and VOP_height code, or based on shape (mask) information obtained by decoding 
code data indicative of shape (mask) information. 

[01 67) As described above, according to the sixth embodiment, even if the coding specifications of a decoder are dif- 
ferent from those of an encoder, code data can be decoded. Further, by deleting object data having the minimum image 
size, selection of object to be deleted is facilitated, and the influence on a decoded image can be suppressed as much 
as possible. 

[0168] Note that in the fifth and sixth embodiments, only one object is deleted, however, two or more object can be 
deleted. Further, it may be arranged such that the user directly designates object(s) to be deleted. 
[01 69] Further, it may be arranged such that the order of deletion is set for the respective objects of image in advance 
by the profile and level input unit 1112. 

Seventh Embodiment 

[01 70] Hereinbelow, a seventh embodiment of the present invention will be described. Note that the general construe- 
35 tion of the moving image processing apparatus according to the seventh embodiment is similar to that in Fig. 29 of the 
fifth embodiment, therefore, an explanation of the construction will be omitted. 

[0171 ] Fig. 33 is a block diagram showing the detailed construction of the profile and level regulator 1 205 according 
to the seventh embodiment of the present invention. In Fig. 33, elements corresponding to those in Fig. 30 have the 
same reference numerals and explanations of the elements will be omitted. In the seventh embodiment, the MPEG4 
40 coding is employed as a moving image coding method, however, any other coding method is applicable as long as it 
encodes a plurality of objects within an image. 

[0172] In Fig. 33, numeral 1 120 denotes an object selection designator which displays a plurality of objects, and in 
which the user's designation of arbitrarily selected objects is inputted; 1 121 , an object selector which selects code data 
of objects to be processed, based on designation from the object selection designator 1 120. and the result of determi- 

45 nation by the profile and level determination 1113; 1122 and 1 124. selectors, controlled by the object selector 1121. 
which switch their input and output; and 1123. an object integrator which integrates a plurality of objects. 
[0173] As in the case of the above -described fifth embodiment, the code data 1 1 01 is inputted into the separator 1 102. 
the profile and level extractor 11 09 and the object counter 1 1 10. The separator 1 1 02 separates the code data 1 101 into 
code data indicative of arrangement information and header information and code data indicative of respective objects. 

so The respective code data are stored into the header memory 1 1 03 and the code memories 1 1 04 to 1 1 08. At the same 
time, the object counter 1110 counts the number of objects included in the code data 1101. 

[0174] Then, as in the case of the fifth embodiment, the profile and level extractor 1 109 extracts information on the 
profile and level from the code data 1101. The decoder status receiver 1111 obtains information on profile and level and 
the like of the decoder 1207. Further, the profile and level are set by the user via the profile and level input unit 1112. 
55 [0175] The profile and level determination unit 1113 compares the profile and level information obtained from the 
decoder 1207 or set by the user, as described above, with the result of extraction by the profile and level extractor 1 1 09. 
If the obtained or set profile and level are higher than or equal to the profile and level extracted from the code data 1101. 
profile and level determination unit 11 13 controls the object selector 1121 to select a path directly connecting the selec- 
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tor 1 122 to the selector 1 124 such that the code data does not pass through the object integrator 1 123. The header 
changer 1 1 15 is not operated. The code data stored in the header memory 11 03 and the code memories 1 104 to 1 108 
are read out in the order of input, and multiplexed by the multiplexer 1116. Thus, the code data 1117 similar to the code 
data 1101 is generated. 

[0176] On the other hand, if the profile and level obtained from the decoder 1207 or set by the user are lower than the 
profile and level extracted from the code data 1101, the profile and level determination unit 1 1 13 inputs the number of 
objects included in the code data 1 101 from the object counter 1 1 10, and compares the number of objects with the 
number of decodable objects, determined from the obtained or set profile and level information. 
[0177] If the number of objects obtained by the object counter 1 1 10 is less than the number of decodable objects the 
code data 1 1 1 7 is generated, as in the case of the above-described case where the obtained or set profile and level are 
higher than or equal to those extracted from the code data 1 101. 

[01 78] On the other hand, if the number of objects obtained by the object counter 1 1 1 0 is greater than the number of 
decodable objects, the number of decodable objects is inputted into the object selector 1 1 21 . The object selector 1 1 21 
displays statuses of the respective objects (e.g. , the image in Fig. 43) , information on the respective objects information 
on the number of integrated objects and the like, on the object selection designator 1 120. The user selects objects to 
be integrated in accordance with these information, and inputs an instruction on the selection into the object selection 
designator 1120. 

[01 79] In the seventh embodiment, as the decoder 1207 performs decoding of Core profile and level 1 , it can decode 
to a maximum of four objects. For example, as the image in Fig. 43 has five objects, two of them are integrated into one 
object, whereby code data decodable by the decoder 1207 can be obtained. Hereinbelow, a case where the user des- 
ignated integration of the object 2003 and the object 2004 in the image in Fig. 43 will be described. 
[0180] When the user designates the objects to be integrated via the object selection designator 1 120, the profile and 
level determmation unit 1113 operates the header changer 1 115 to change the content of PLI in correspondence with 
the decoder 1207. generate header information on the new object obtained by integration and delete header informa- 
tion on the objects deleted by the integration, based on the result of selection by the object selector 1121 More specif- 
ically, arrangement information of the new object obtained as a result of integration is generated and arrangement 
information of the original objects 2003 and 2004 are deleted, based on the arrangement information of the objects 
2003 and 2004. Then, the size of the object obtained by the integration or other information are generated as header 
information and header information of the original objects 2003 and 2004 are deleted, based on the header information 
of the objects 2003 and 2004. 

[0181] The object selector 1121 controls the input/output of the selectors 1 122 and 1 124 so as to perform integration 
processing by the object integrator 1 1 23 with respect to code data of the objects 2003 and 2004, and to avoid process- 
ing by the object integrator 1 123 with respect to other code data. 

[01 82] Then, contents of the header changer 1 1 1 5 and the code memories 1 1 04 to 1 1 06 holding the code data of the 
objects 2000 to 2002 are read out in the order of input, and multiplexed by the multiplexer 1 116 via the selectors 1122 
and 1 124. On the other hand, the contents of the code memories 1 107 and 1 108 holding the code data of the objects 
2003 and 2004 to be integrated are inputted via the selector 1 122 to the object integrator 1 123. 

[Object Integrator] 

[0183] Fig. 34 is a block diagram showing the detailed construction of the object integrator 1 123. In Fig. 34, numerals 
1050 and 1051 denote code memories respectively for storing code data of objects to be integrated; 1052 and 1054, 
selectors which switch input/output for respective objects; 1053, an object decoder which decode code data and repro- 
duces an image of an object; 1055 and 1056, frame memories for storing reproduced images for respective objects; 
1057, a synthesizer which synthesizes objects in accordance with arrangement information of objects to be integrated 
stored in the header memory 1 103; and 1058. an object encoder which encodes image data obtained by synthesizing 
and outputs the image data. 

[01 84] Hereinbelow, the operation of the object integrator 1 1 23 will be described in detail. The code data of the objects 
2003 and 2004 to be integrated are stored into the code memories 1050 and 1051. First, the selector 1052 selects an 
input on the code memory 1 050 side, and the selector 1054, an output on the frame memory 1 055 side. Thereafter, the 
code data is read out from the code memory 1050, and decoded by the object decoder 1053. Then image information 
of the object 2003 is written via the selector 1054 into the frame memory 1055. The image information of the object 
comprises image data indicative of a color image and mask information indicative of a shape. Then, the input and out- 
put of the selector 1 052 and 1 054 are switched to the opposite sides, and similar processing is performed, whereby the 
image information of the object 2004 is stored into the frame memory 1056. 

[01 85] The synthesizer 1 057 obtains location information and size information of the objects 2003 and 2004 from the 
header, memory 1 1 03, and obtains the image size of the new object obtained by object synthesizing and relative loca- 
tions of the original objects 2003 and 2004 in the new object. Then, the image information in the frame memories 1055 
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and 1056 are read out, and the color image information and the mask information are respectively synthesized. Fig. 37 
is shows the result of synthesizing of color image information. Fig. 38 shows the result of synthesizing of mask informa- 
tion. The object encoder 1058 encodes these color image information and mask information in accordance with the 
MPEG4 object coding. Then, the object integrator 11 23 outputs the encoded information. 

[0186] The code data outputted from the object integrator 1 1 23 is multiplexed with other code data by the multiplexer 
1116 via the selector 1 124, thus the code data 1 1 17 is obtained. Fig. 35 shows a bit stream of the code data 1 1 17. Fig. 
35 shows the result of integration processing according to the seventh embodiment with respect to the code data 1 101 
in Fig. 31 A. In Fig. 35. the bit stream has arrangement information y including arrangement information of the newly 
obtained object as the result of synthesizing. VOSSC, Visual Object data y-1. Y-2 and y-3, and VOSEC. The Visual 
Object data y-1 . y-2 and y-3 are obtained by object integration regulation with respect to the original Visual Object data 
a-1 . a-2 and a-3 shown in Fig. 31 A. For example, the Visual Object data y-1 . following Visual Object SC, comprises PL1- 
y indicative of profile and level appropriate to the decoder 1207, VO data A, VO data B and VO data C as respective 
code data of the objects 2000 to 2002, and code data VO data G obtained by integrating the objects 2003 and 2004. 
[01 87] The code data 1117 obtained as above is stored into the storage device 1 206. or decoded by the decoder 1 207 
and reproduced as an image as shown in Fig. 43 and displayed on the display unit 1208. 

[01 88] Note that in the seventh embodiment, the user selects and designates objects to be integrated within an image 
by the object selection designator 1120. however, the present invention is not limited to this example. For example, it 
may be arranged such that the integration order is set for objects of the image in advance by the object selection des- 
ignator 1 120, then if the number of objects decodable by the decoder 1207 is less than the number of objects of the 
image and object integration is required, object integration is automatically performed in accordance with the set order. 
[0189] As described above, according to the seventh embodiment, even if profile and/or level of a decoder are differ- 
ent from those of an encoder, code data can be decoded. Further, by integrating objects and decoding the integrated 
object, loss of decoded object can be prevented. 

[01 90] Further, the object integration processing can be performed in incremental order of code length or image size 
by providing the code length comparator 1114 and the size comparator 1118 shown in the fifth and sixth embodiments 
in place of the object selection designator 1 120 and the object selector 1 121 for controlling the object integrator 1 123. 
[0191 ] Fig. 36 is a block diagram showing the construction of the object integrator 1 1 23 according to a modification 
of the seventh embodiment. In Fig. 36. elements corresponding to those in Fig. 34 have the same reference numerals 
and explanations of the elements will be omitted. The construction of Fig. 36 is characterized by further comprising a 
code length counter 1059. The code length counter 1059 counts code lengths of code data of respective objects prior 
to integration, and parameters (e.g., quantization parameters or the like) of the object encoder 1058 is controlled such 
that the code length of output from the object encoder 1058 is the same as the counted result. Thus the objects can be 
synthesized without increasing the total code length. 

35 Eighth Embodiment 

[0192] Hereinbelow, an eighth embodiment of the present invention will be described. As in the case of the above- 
described seventh embodiment, object integration processing is performed in the eighth embodiment. Note that the 
general construction of the moving image processing apparatus of the eighth embodiment, and the detailed construc- 
40 tion of the profile and level regulator 1205 are the same as those in Fig. 33. therefore explanations of the apparatus and 
the construction of the profile and level regulator will be omitted. 

[0193] Fig. 39 is a block diagram showing the detailed construction of the object integrator 1123 according to the 
eighth embodiment of the present invention. In Fig. 39. elements corresponding to those in Fig. 34 have the same ref- 
erence numerals and explanations of the elements will be omitted. 

45 [01 94] In Fig. 39. numerals 1 060 and 1 061 denote separators which separate input code data into code data on mask 
information indicative of shape and code data indicative of color image information and output the separated data. 
Numeral 1062 to 1065 denote code memories. The code data indicative of color image information is stored into the 
code memories 1062 and 1064. and the code data on mask information is stored into the code memories 1063 and 
1065, for respective objects. Numeral 1066 denotes a color image information code synthesizer which synthesizes the 

so code data indicative of color image information in the form of code data; 1067. a mask information code synthesizer 
which synthesizes the code data indicative of mask information in the form of code data; 1 068. a multiplexer which mul- 
tiplexes code outputted from the color image information code synthesizer. 1066 and the mask information code synthe- 
sizer 1067. 

[01 95] Hereinbelow. the operation of object integrator 1 1 23 according to the eighth embodiment will be described in 
55 detail. As in the case of the seventh embodiment, the code data of the objects 2003 and 2004 are stored respectively 
into the code memories 1050 and 1051 . The code data of the object 2003 stored in the code memory 1050 is read out 
in frame units (VOP units), separated by the separator 1060 into code data of color image information and code data of 
mask information, and the respective code data are stored into the code memories 1062 and 1063. Similarly, code data 
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of color image information and code data of mask information of the object 2004 are stored into the code memories 
1064 and 1065. 

[01 96] Thereafter, the color image information code synthesizer 1 066 reads the color image information code data 
from the code memories 1062 and 1064. Further, as in the case of the seventh embodiment, the color image informa- 
tion code synthesizer 1066 obtains location information and size information of the objects 2003 and 2004 from the 
header memory 1103. and obtains the image size of synthesized new object and respective relative locations of the 
original objects 2003 and 2004 in the new object. That is, the color image information code synthesizer 1066 performs 
synthesizing on the assumption that if these color image information code data are synthesized and decoded, an image 
as shown in Fig. 37 can be obtained as one object. 

[01 97] Note that the MPEG4 coding method has a slice data structure to define a plurality of macro blocks as a cluster 
of blocks in a main scanning direction. Fig. 40 shows an example of the slice structure applied to the objects in Fig. 37. 
In Fig. 40, an area in a bold frame is defined as one slice. In each slice, the head macro block is hatched. 
[01 98] The color image information code synthesizer 1066 performs reading in a rightward direction (main scanning 
direction) as shown in Fig. 40, sequentially from an upper left macro block data of the image, to be obtained as a result 
of synthesizing. That is, among the code data of the object 2003, code data corresponding to the head macro block of 
the head slice is read from the code memory 1062 first. The header information of the slice is added to the read code 
data, and the code data of the head macro block is outputted. Then, the code data corresponding to the macro block 
on the right of the head macro block is read and outputted. In this manner, the read and output operations are sequen- 
tially repeated to the slice. 

[01 99] Note that a portion where data has been newiy generated between the objects 2003 and 2004 is considered 
as a new slice. As this portion is not displayed even if decoded with mask information, appropriate pixels are provided 
to cover the portion. That is. such portion comprises only DC component of the last macro block including an object. As 
the DC difference is "0", and all the AC coefficients are "0", no code is generated. 

[0200] Then, as it is considered that a new slice has started on the edge of the object 2004, a hatched macro block 
in Fig. 40 is regarded as the head of new slice, and the header information of the slice is added to the block. In this case, 
as the address of the head macro block is an absolute address, the address is converted to a relative address from the 
macro block including the previous object. Note that in the macro block, if DC component or the like is predicted by refer- 
ring to another macro block, that portion is re-encoded, then code data of the macro block is sequentially outputted in 
the rightward direction. That is. the slice header is added on the edge of object, and the prediction of the slice head 
macro block is replaced with initialized code. The obtained code is outputted to the multiplexer 1068. 
[0201] In parallel to the operation of the color image information code synthesizer 1066, the mask information code 
synthesizer 1067 reads the code data of the mask information from the code memories 1063 and 1065. Then, the mask 
information code synthesizer 1067 obtains location information and size information of the objects 2003 and 2004 from 
the header memory 1 103, and obtains the image size of a synthesized new object and relative locations of the original 
objects 2003 and 2004 in the new object. Then, by decoding and synthesizing the input code data of the mask informa- 
tion, the mask information code synthesizer 1 067 obtains mask information as shown in Fig. 38. The mask information 
code synthesizer 1067 encodes the mask image by an arithmetic encoding as the MPEG4 shape information coding 
method. The obtained code is outputted to the multiplexer 1 068. 

[0202] Note that the mask information coding is not limited to the MPEG4 arithmetic coding method. For example, in 
the result of synthesizing of mask information code data, as the zero-run between object edges is merely lengthened, 
the synthesizing can be made only by replacing code representing the zero-run length without decoding by the mask 
information code synthesizer 1067, by employing zero-run coding or the like, used in a facsimile apparatus. Generally, 
even when mask information is encoded by the arithmetic or another coding, the code length is merely slightly changed. 
[0203] The multiplexer 1 068 multiplexes the code data on the synthesized color image information and the code data 
of the mask information, as code data of one object. The subsequent processing is similar to that in the above- 
described seventh embodiment. The multiplexer 1116 multiplexes the code data with other code data and outputs the 
data. 

[0204] As described above, according to the eighth embodiment, even in a case where the profile and/or level of an 
encoder are different from those of a decoder, code data can be decoded. Further, as objects are integrated in the form 
of code data, loss of object in decoded image data can be prevented only by adding header information. 
[0205] Further, in the object integration processing according to the eighth embodiment, a newly added header can 
be obtained by a slight amount of calculation, and further, code change is limited to the head block of a slice. Accord- 
ingly, the object integration processing can be performed at a speed higher than that in the object integration processing 
by decoding and re-encoding according to the seventh embodiment. 

Ninth Embodiment 

[0206] Hereinbelow, a ninth embodiment of the present invention will be described. In the ninth embodiment, object 
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integration processing is performed, as in the case of the above-described seventh embodiment,. Note that the general 
construction of the moving image processing apparatus according to the ninth embodiment is similar to that in Fig. 29 
of the fifth embodiment, therefore, an explanation of the construction will be omitted. 

[0207] Fig. 41 is a block diagram showing the detailed construction of the profile and level regulator 1 205 according 
5 to the ninth embodiment of the present invention. In Fig. 41 . elements corresponding to those of the seventh embodi- 
ment in Fig. 33 have the same reference numerals, and explanations of the elements will be omitted. In the ninth 
embodiment, the MPEG4 coding is employed as a moving image coding method, however, any other coding method is 
applicable as long as it encodes a plurality of objects within an image. 

[0208] In Fig. 41, numeral 1170 denotes an object arrangement information determination unit which determines 
io objects to be integrated. 

[0209] As in the case of the seventh embodiment, the profile and level determination unit 1113 compares the profile 
and level information of the decoder 1207 with those of the code data 1 101 . Even if the profile and level of the decoder 
1 207 are higher than or equal to. or lower than those of the code data 1 101 , the code data 1 1 1 7 is generated in a similar 
manner to that of the seventh embodiment as long as the number of objects obtained by the object counter 1 1 1 0 is the 

is number decodable by the decoder 1 207. 

[0210] On the other hand, if the number of objects obtained by the object counter 1 1 10 is greater than the number 
decodable by the decoder 1207. the decodable number of objects is inputted into the object arrangement information 
determination unit 1 170. As in the case of the seventh embodiment the maximum number of objects decodable by the 
decoder 1207 is four. Accordingly, in an image having five objects as shown in Fig. 43. decodable code data can he 

20 obtained by integrating two objects. 

[021 1 ] The object arrangement information determination unit 1 1 70 extracts location information and size information 
of the respective objects from the header memory 1 103, and determines two objects to be integrated based on the fol- 
lowing conditions. Note that condition (1) is given higher priority to condition (2). 



25 (1) One object is included in the other object 

(2) The distance between both objects is the shortest 

[021 2] In the image shown in Fig. 43. the objects 2001 to 2004 are included in the object 2000. Accordingly, the object 
arrangement information determination unit 1 1 70 determines the object 2000 and the object 2001 as objects to be inte- 
30 grated. 

[021 3] When the objects to be integrated have been determined, the profile and level determination unit 1113 oper- 
ates the header changer 1 1 15 to change and encode the content of the PLI in accordance with the decoder 1207, and 
generate header information on a new object obtained by object integration and delete the header information on the 
integrated objects, as in the case of the seventh embodiment, based on the result of determination by the object 

35 arrangement information determination unit 1170. More specif icatly. arrangement information on the new object 
obtained by object integration is generated, based on the arrangement information of the objects 2000 and 2001. and 
arrangement information of the original objects 2000 and 2001 are deleted. Then, the image size information or other 
information of the object obtained by the integration is generated as header information, based on the header informa- 
tion of the objects 2000 and 2001 , and the header information of the original objects 2000 and 2001 are deleted. 

40 [021 4] The object arrangement information determination unit 1 1 70 controls input/output of the selectors 1 1 22 and 
1 124 so as to perform integration processing on the code data of the objects 2000 and 2001 by the object integrator 
1 123, on the other hand, so as not to pass the other code data through the object integrator 1 123. 
[021 5] Then, the contents of the header changer 1 1 1 5 and the code memories 1 106 to 1 108 holding the code data 
of the objects 2002 to 2004 are read out sequentially in the order of input, and inputted via the selectors 1 1 22 and 1 1 24 

as into the multiplexer 1 1 1 6. On the other hand, the contents of the code memories 1 1 04 and 1 1 05 holding the code data 
of the objects 2000 and 2001 to be integrated are integrated by the object integrator 1 123, and inputted into the multi- 
plexer 1116. The multiplexer 1116 multiplexes these code data, thus generates the code data 1117. Note that the inte- 
gration processing by the object integrator 11 23 is realized in a similar manner to that in the above-described seventh 
embodiment or eighth embodiment. 

so [021 6] Fig. 42 shows a bit stream of the code data 1117 according to the ninth embodiment. Fig. 42 shows the result 
of integration processing of the ninth embodiment performed on the code data 1 101 as shown in Fig. 31 A. In Fig. 42. 
arrangement information 6 including arrangement information of the newly obtained object is provided at the head. 
Then VOSSC. Visual Object data 8-1 . 5-2. 8-3, and VOSEC follow. The Visual Object data 8-1 . fi-2, 8-3 are obtained by 
performing object integration regulation on the original Visual Object data a-1 . a-2. and a-3 in Fig. 31 A. For example. 

55 the Visual Object data 8-1 comprises Visual Object SC. then PLI-8 indicative of profile and level appropriate to the 
decoder 1207. VO data H as code data obtained by integrating the objects 2000 and 2001 , and VO data C. VO data D 
and VO data E as code data of the objects 2002 to 2004. 

[021 7] The code data 1117 obtained as above is stored into the storage device 1 206. or decoded by the decoder 1 207 
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and reproduced as an image as shown in Fig. 43, and displayed on the display unit 1208. 

[021 8] Note that in the ninth embodiment, as in the cases of the fifth and sixth embodiments, code lengths of respec- 
tive objects, object sizes and the like may be added to the conditions for determining objects to be integrated. 
[021 9] As described above, according to the ninth embodiment, even if the profile and/or level of an encoder are dif- 
5 ferent from those of decoder, code data can be decoded. Further, loss of decoded object can be prevented while sup- 
pressing the amount of code changed by integration, by integrating the objects based on the location relation among 
the objects. 

[0220] In the ninth embodiment, objects to be integrated are determined based on the location relation among the 
objects. The determination according to the ninth embodiment may be employed in the above-described fifth and sixth 
10 embodiments. That is, objects to be deleted can be selected based on location information of objects. 

[0221] Note that in the seventh to ninth embodiments, two objects are integrated and one object is generated. How- 
ever, three or more objects, or two or more sets of objects may be integrated. 

[0222] Note that the arrangement of the code memories 1 104 to 1 108 and the header memory 1 103 is not limited to 
that shown in Fig. 41 . More code memories can be provided, or one memory may be divided into a plurality of areas. 
15 Further, a storage medium such as a magnetic disk may be employed. 

[0223] Further, the selection of objects to be deleted or integrated may be determined based on the combination of a 
plurality of conditions such as sizes and code lengths of objects, location relation among the objects and user's instruc- 
tion. 

[0224] Further, in a case where the fifth to ninth embodiments are applied to an image editing apparatus, even if the 
20 number of objects changes due to editing processing, the output from the apparatus can be adjusted to an arbitrary pro- 
file and/or level. 

[0225] As described above, according to the fifth to ninth embodiments, code data encoded for a plurality of image 
information (objects) can be decoded by decoders of arbitrary specifications. Further, the number of objects included in 
the code data can be regulated. 

25 

Tenth Embodiment 
[Construction] 

30 [0226] Fig. 45 is a block diagram showing the construction of the moving image processing apparatus according to a 
tenth embodiment of the present invention. In the tenth embodiment, the MPEG4 coding is employed as a moving 
image coding method. Note that the coding method is not limited to the MPEG4 coding, but any other coding method 
is applicable as long as it encodes a plurality of objects within an image. 

[0227] In Fig. 45, numerals 2201 and 2202 denote storage devices holding moving image code data. The storage 
35 devices 2201 and 2202 respectively comprise a magnetic disk, an magneto-optical disk, a magnetic tape, a semicon- 
ductor memory or the like. Numeral 2203 denotes a TV camera which obtains a moving image and outputs a digital 
image signal; 2204, an encoder which performs coding by the MPEG4 coding method; 2205, a communication line of 
a local area network (LAN), a public line, a broadcasting line or the like; 2206, a communication interface which receives 
coded data from the communication line 2205; and 2207, an editing operation unit which displays image editing condi- 
40 tion. The user inputs editing instruction from the editing operation unit 2207. Further, numeral 2208 denotes an image 
editing unit characteristic of the present embodiment; 2209, a storage device for storing output from the image editing 
unit 2208; 2210, a decoder which decodes code data of a moving image encoded by the MPEG4 coding; 221 1 , a dis- 
play unit which displays a moving image decoded by the decoder 2210. 

45 [Image Editing] 

[0228] Hereinbelow, image editing processing of the present embodiment will be described using a specific image as 
an example. 

[0229] Image data, encoded by the MPEG4 coding of Core profile and level 2 at a bit rate of 384 kbps, is stored into 
so the storage device 2201 . Fig. 46A shows an example of the image stored in the storage device 2201 . Fig. 50A shows 
the code data of the image. In the image of Fig. 46A, a background object 2300 includes objects 2304 and 2305 repre- 
senting men. In Bg. 50 A. code data of the background object 2300 is VO data A-1 -1 , and code data of the men objects 
2304 and 2305 are VO data A-1 -2 and VO data A-1 -3. 

[0230] Image data, encoded by the MPEG4 coding of Core profile and level 1 at a bit rate of 200 kbps, is stored into 
55 the storage device 2202. Fig. 46B shows an example of the image stored in the storage device 2202. Fig. 50B shows 
the code data of the image. In the image of Fig. 46B. a background object 2301 includes objects 2306 and 2307 repre- 
senting a man and a woman. In Fig. SOB, code data of the background object 2301 is VO data B-1 -1 , and code data of 
the man and woman objects 2306 and 2307 are VO data B-1 -2 and VO data B-1 -3. 
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[0231 ] In a case where the TV camera 2203 obtains an image as shown in Fig. 46C and the encoder 2204 encodes 
the image data by the MPEG4 coding of Simple profile and level 1 at a bit rate of 32 kbps. as a new object is not 
extracted from the obtained image, the entire image is handled as one object 2302. Accordingly, as shown in Fig. 50C ( 
code data of the image comprises VO data C-1 -1 . code data of one object 2302. 

£ [0232] Further, in a case where an image as shown in Fig. 46D is encoded by the MPEG4 coding of Simple profile 
and level 2 and inputted from the communication line 2205 via the communication interface 2206, a background object 
2303 in the image in Fig. 46D includes objects 2308 and 2309 representing a woman and a man. Fig. SOD shows code 
data of the image, in which code data of the background object 2303 is VO data D-1-1. code data of the man and 
woman objects 2308 and 2309 are VO data D-1-2 and VO data D-1-3. 

io [0233] Note that to simplify the explanation, the sizes of all the above-described images (Figs. 46A to 46D) are defined 
with QCIF (Quarter Common Intermediate Format). 

[Image Editing Unit] 

15 [0234] All the code data are inputted into the image editing unit 2208. Fig. 47 is a block diagram showing the con- 
struction of the image editing unit 2208. In Fig. 47, numerals 2101 to 2104 denote system code memories for storing 
system-related code data for respective inputs; 21 05 to 2 108, video code memories for storing moving image code data 
for respective inputs; 2109, a video decoder which decodes moving image code data to reproduce objects; and 21 10, 
a system decoder which decodes the system code data to reproduce object arrangement information and the like. 

so [0235] The results of decoding are outputted to the editing operation unit 2207, and the respective objects are dis- 
played in accordance with the arrangement information. The editing operation unit 2207 newly sets display timing, 
speed and the like, in accordance with designation of arrangement of these objects, size change, deformation and the 
like, instructed by the user. 

[0236] Numeral 2111 denotes a system code synthesizer which synthesizes system code; 21 12, a header processor 
25 which synthesizes or changes headers of video code; 21 13, a selector which arbitrarily selects one of outputs from the 
video code memories 2105 to 2 108 and outputs the selected output; 21 14. a multiplexer which multiplexes outputs from 
the system code synthesizer 21 1 1 . the header processor 21 1 2 and the selector 21 1 3 to generate code data. 
[0237] In the image editing unit 2208, respective outputs from the storage devices 2201 and 2202, the encoder 2204 
and the communication interface 2206 are separated into system code data and moving image code data. The system 
30 code data are stored into the system code memories 2101 to 2104, and the moving image code data are stored into 
the video code memories 2105 to 2108. 

[0238] When the respective code data have been stored, the video decoder 2109 and the system decoder 21 10 
decode the respective data, and output the decoded data to the editing operation unit 2207. In the editing operation unit 
2207, the user sets settings of deletion/holding objects, change of arrangement, moving image start timing, frame rate 
35 and the like. The video decoder 2209 and the system decoder 2110 arbitrarily perform decoding in accordance with the 
editing operation. 

[0239] Fig. 48 shows an example of an image synthesized from the images shown in Figs. 46 A to 46D. That is, a new 
image 2320 is generated by editing and synthesizing the four images. The size of the image 2320 is defined with CIF 
format because the QCIF four images are synthesized without overlapping with each other. In the image 2320, the 
40 background object 2300, the object 2302, the background objects 2303 and 2301 are arranged, from an upper left posi- 
tion in a clockwise manner. Further, the men objects 2304 and 2305 are moved horizontally in rightward direction 
(edited). The object 2308 is enlarged and moved onto the background object 2300 (edited). 

[0240] The system code synthesizer 2 1 1 1 reads out the system code data from the system code memories in accord- 
ance with the results of synthesizing, then generates new system code data with arrangement information correspond- 

45 ing to these deformation and movement, and outputs the new system code data to the multiplexer 21 14. 

[0241 ] Next, the changing condition accompanying synthesizing of respective objects will be described below. 
[0242] First, regarding the background object 2300. coordinates, start timing and the like have not been changed. 
Regarding the background object 2301, its coordinates (0,0) has been changed to (0,144). Regarding the object 2302. 
its coordinates (0,0) has been changed to (176,0). Regarding the background object 2303. its coordinates (0,0) has 

so been changed to (176,144). 

[0243] Regarding the men objects 2304 and 2305, coordinate values for the rightward movement have been added 
to their coordinates. Regarding the objects 2306 and 2307. the coordinates have been changed in correspondence with 
the change of the coordinates of the background object 2301 from (0.0) to (0.144). so as to move the absolute positions 
downward by "144". Regarding the object 2308, new coordinates have been generated based on the expansion desig- 

55 nation (magnification ratio) and a new distance from the origin (0.0). Regarding the object 2309. its coordinates have 
been changed in correspondence with the change of the coordinates of the background object 2303 from (0.0) to 
(176,144), so as to move the absolute position rightward by "176" and downward by "144". 

[0244] Note that in horizontal movement of object, the system code synthesizer 21 11 merely adds the amount of 
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movement to the coordinates of display position with respect to the code data of the object, however, in expansion or 
deformation processing, generates commands corresponding to those processing and newly performs coding. Note 
that system code in the MPEG4 standard is similar to the CG language VRML, therefore, detailed commands are 
approximately similar to those in the VRML or ISO/IEC 14496-1 . 

5 [0245] On the other hand, the header processor 2112 generates a new header in correspondence with the results of 
editing of the system code data. Fig. 49 is a block diagram showing the detailed construction of the header processor 
2112. In Fig. 49, numeral 2120 denotes a separator which separates input header information for respective codes and 
determine output destinations; 2121. a profile determination unit; 2122, an object number determination unit; 2123, a 
bit rate determination unit; and 2124, a profile determination unit which determines a profile. 

10 [0246] In the header processor 21 12, the separator 2120 extracts PLI code, VOSC and bitrate code from header infor- 
mation of the respective objects, from video code memories 21 05 to 2108, and inputs the extracted code into the profile 
determination unit 2121, the object number determination unit 2122 and the bit rate determination unit 2123. The profile 
determination unit 2121 decodes the PLI code and detects the highest profile and level from profiles and levels of 
images to be synthesized. The object number determination unit 2122 counts the number of objects included in the 

is code data by counting the VOSC. The bit rate determination unit 2123 detects the respective bit rates by decoding the 
bitrate code, and obtains the total sum of the bit rates. The outputs from the respective determination units are inputted 
into the profile determination unit 21 24. 

[0247] The profile determination unit 2124 determines profile and level satisfying the highest profile, the number of 
objects and bit rate, by referring to the profile table as shown in Fig. 28. In the present embodiment, the highest profile 
20 of the four images to be synthesized is Core profile and level 2, the number of objects of the synthesized images is 10, 
and the total sum of the bit rates is 684 kbps. Accordingly, the profile and level satisfying these conditions is, according 
to the profile table. Main profile and level 3. The profile determination unit 2124 generates new PLI code based on Main 
profile and level 3, and outputs the PL! code. 

[0248] The multiplexer 2114 multiplexes the system code data generated by the system code synthesizer 21 1 1 and 
25 the code data of moving image. The moving image code data is reproduced by reading the code, where profile- related 
code or the like is corrected, from the header processor 2112, or arbitrarily reading the code data stored in the video 
code memories 2105 to 2108, and multiplexing the read data. Then, the multiplexed code data is outputted to the stor- 
age device 2209 and the decoder 2210. 

30 [Processing Procedure] 

[0249] Fig. 50E shows code data obtained as a result of multiplexing by the multiplexer 21 14. It is understood from 
Fig. 50E that all the code data shown in Figs. 50A to SOD are synthesized, i.e., all the objects in Figs. 46A to 46D are 
included. Note that in the multplexed code data, user data may be positioned prior to the code data of the respective 

35 objects, or intensively positioned in a predetermined position within the code data. 

[0250] Fig. 51 is a flowchart showing image processing according to the present embodiment. When the apparatus 
has been started, code data of images are inputted from the respective image input means (storage devices 2201 and 
2202, encoder 2204 and communication interface 2206), and stored into the code memories 2101 to 2104 and 2105 to 
2108 (step Si 01). Then, the code data are respectively decoded, and images represented by the decoded data are pre- 

40 sented to the user (step S102). Thereafter, the results of the user's edition at the editing operation unit 2207 is obtained 
(step S103), and the system code is changed (step S104) in accordance with the obtained results of editing. Further, 
the header of moving image code data are changed in accordance with the profile and level, the number of objects, the 
bit rate and the like, so as to generate new code (step S105). Then, in the multiplexer 21 14, the system code data and 
video code data are multiplexed and outputted (step S106). 

45 [0251] As code data synthesized by the image editing unit 2208 is inputted into the decoder 2210, the decoder 2210 
easily detects the scale of input code data to be decoded, the number of necessary decoders and the like. Accordingly, 
it can be easily determined whether or not decoding is possible without actually decoding the code data. For example, 
even if it is determined that decoding is impossible, the code data can be temporarily stored into the storage device 
2209 and decoded when a necessary number of decoders are provided. ' 

so [0252] Note that the arrangement of the system code memories 21 01 to 2104 and the video code memories 21 05 to 
2108 of the present embodiment is not limited to that in Fig. 47. however, more code memories may be provided, or one 
memory may be divided into a plurality of areas. Further, a storage medium such as a magnetic disk may be employed. 
[0253] According to the present embodiment, when code data of different profiles and/or levels are synthesized, pro- 
file and level are re-defined. Since the scale of code data to be inputted, a necessary number of decoders and the like 

55 are obtained in advance, in the decoder 2210, it can he easily determined whether or not decoding is possible. 
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Eleventh Embodiment 

[0254] Hereinbelow. an eleventh embodiment of the present invention witl he described. Note that the general con- 
struction of the moving image processing apparatus of the eleventh embodiment is similar to that of the above- 
£ described tenth embodiment in Fig. 45, therefore, an explanation of the construction will be omitted. In the eleventh 
embodiment, the user designates an arbitrary profile using the editing operation unit 2207, and the image editing unit 
2208 generates code data based on the designated profile. 

[Construction] 

10 

[0255] Fig. 52 is a block diagram showing the detailed construction of the image editing unit 2208 according to the 
eleventh embodiment. In Fig. 52, elements corresponding to those in Fig. 47 of the tenth embodiment have the same 
reference numerals, and explanations of the elements will be omitted. In the eleventh embodiment, the MPEG4 coding 
method is employed as a moving image coding method, however, any other coding method is applicable as long as it 

is encodes a plurality of objects within an image. 

[0256] Numeral 2130 denotes a profile controller which performs various controls to synthesize input plural image 
data in correspondence with a profile designated from the editing operation unit 2207, 21 31 , a system code synthesizer 
which synthesises system code; 2132, a header processor which synthesizes and changes header of video code;- 
2134, a code length regulator which regulates code lengths of respective objects; 2136, an integration processor which 

20 performs integration processing on objects; and 2133, 2135 and 2137, selectors which switch respective input/output 
in accordance with an instruction from the profile controller 2130. 

[0257] As in the case of the above-described tenth embodiment, the code data inputted from the storage devices 
2201 , 2202, the encoder 2204 and the communication interface 2206 are separated into system code data and moving 
image code data, and stored into the system code memories 2101 to 2104 and the video code memories 2105 to 2108. 

25 [0258] Note that in the eleventh embodiment, the code data inputted from the storage devices 2201 and 2202. the 
encoder 2204 and the communication interface 2206 are the same as those in the above-described tenth embodiment. 
Accordingly, the respective images are the same as those in Figs. 46A to 46D, and the code data in Figs. 50A to SOD 
are obtained by encoding the respective images. Note that in the eleventh embodiment, code data (VO data A) of Core 
profile and level 2 and at a bit rate of 1024 kbps is inputted from the storage device 2201 . Similarly, code data (VO data 

30 B) of Core profile and level 1 and at a bit rate of 384 kbps is inputted from the storage device 2202. Similarly, code data 
(VO data C) of Simple profile and level 3 and at a bit rate of 384 kbps is inputted from the encoder 2204, and code data 
(VO data D) of Core profile and level 2 and at a bit rate of 768 kbps is inputted from the communication interface 2206. 
[0259] In this embodiment, these code data have information unique to the respective objects as user data. The 
objects in the eleventh embodiment are "people", "background" and "non-cut-out screen image". As user data of a 

35 "man" object, information indicating that the type of objects is "man - , personal information of the man (sex, age, profes- 
sion and the like), further, action of the man in the image (e.g., the men objects 2304 and 2305 are discussing, the man 
object 2307 is giving an injection to the girl object 2306). These object-unique information are utilized upon editing oper- 
ation such as object search. 

[0260] When the respective code data have been stored into the code memories, the video decoder 2109 and the 
40 system decoder 21 10 respectively decode the code data and output the decoded data to the editing operation unit 
2207. At the editing operation unit 2207, the user operates settings such as selection of deletion/holding objects, 
change of arrangement, moving image start timing and frame rate, thus, the synthesized image 2320 as shown in Fig. 
48 is obtained, as in the case of the tenth embodiment. 

45 [Setting of Profile and Level] 

[0261 ] As described above, in the eleventh embodiment the user can arbitrarily set the profile and level of code data 
to be outputted, from the editing operation unit 2207. Accordingly, when the generated code data is delivered by broad- 
casting or the like, the user can adjust the profile and level of the code data to those of a decoder to receive the code 
so data. Hereinbelow, a case where the user has designated Core profile and level 2 at the editing operation unit 2207 will 
be described. 

[0262] The user's designation of profile and level is inputted, with the results of editing, into the profile controller 21 30. 
The synthesized image 2320 shown in Fig. 48 includes 10 objects, and the total sum of the bit rates is 2560 kbps. Fur- 
ther, in Core profile and level 2 designated by the user, the maximum number of objects is 8. and the maximum bit rate 
55 is 2048 kbps. according to the profile table in Fig. 28. To perform decoding of the designated profile at the designated 
level, the number of objects of the synthesized image must be reduced by two. and the bit rate must be controlled to 
2048 kbps. 

[0263] The profile controller 2130 reduces code length of code data based on the following conditions in the numerical 
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priority order. 

(1) Code length is reduced from the highest profile level 

(2) Code length is reduced from a highest bit rate 
s (3) All the code lengths are reduced 

[0264] Hereinbelow. the bit rate of the VO data A is reduced from 1 024 kbps to 512 kbps by reducing the code length 
of the VO data A based on these conditions. 

[0265] Further, to reduce the number of objects, two objects may be synthesized into one object, for example. In the 
10 eleventh embodiment, objects tohe integrated are determined from a plurality of objects, by referring to node informa- 
tion in the system code stored in the system code memories 2101 to 2104. That is, parent-child relation of nodes are 
referred to. and objects having the same parent are integrated. 

[0266] Hereinbelow, the object integration processing according to the eleventh embodiment will be described. 
[0267] Figs. 53A to 53D show node statuses of the respective objects in the eleventh embodiment. Fig. 53 A shows a 

t5 node relation of the image in Fig. 46A. The code data is divided into the background 2300 and People node represent- 
ing people, further, the People node is a parent of the "men" objects 2304 and 2305. Similarly, Fig. 53B shows a node 
relation of the image in Fig. 46B; Fig. 53C. a node relation of the image in Fig. 46C; and Fig. 53D, a node relation of the 
image in Fig. 46D. That is, in Fig. 53A, the "men" objects 2304 and 2305 are connected to the People node; in Fig. 53B, 
the "girl" object 2306 and the "doctor" object 2307 are connected to the People node; in Fig, 53 D, the "woman" object 

20 2308 and the "man" object 2309 are connected to a dancer node. 

[0268] Accordingly, in the eleventh embodiment, the objects connected to the People and dancer nodes indicative of 
people are determined as objects to be integrated for respective images. That is, in the image in Fig. 46A, the objects 
2304 and 2305 are integrated. Similarly, in the image in Fig. 46B, the objects 2306 and 2307 are integrated; in the 
image in Fig. 46D, the objects 2308 and 2309 are integrated. By this integration, the number of objects in the synthe- 

25 sized image becomes seven, and the number of objects satisfies Core profile and level 2. 

[0269] The profile controller 2130 instructs the system code synthesizer 2131 to newly reproduce the arrangement 
information of the respective objects after the object integration. The system code synthesizer 2131 generates system 
code data in the state where the objects are integrated, as in the case of the tenth embodiment 
[0270] At the same time, the profile controller 2130 instructs the header processor 2132 to newly generate header 

30 information of the respective objects after the object integration. That is, the size of image is changed to CIF(352x288), 
the bit rate is set to 2048 kbps, and the PLI code is set to Core profile and level 2. Further, code such as VOL_width, 
VOL_height, VOP_width, VOP_heigh and bitrate of the integrated objects are corrected. 

[0271 ] The selector 21 33 switches a data path so as to pass the object of the image in Fig. 46 A (VO data A) through 
the code length regulator 2134, and not to pass the other objects through the code length regulator 2134, under the 

35 control of the profile controller 21 30. 

[0272] Fig. 54 is a block diagram showing the construction of the coding length regulator 2134. An object decoder 
2141 decodes input video code data, and an object encoder 2142 encodes the decoded data using quantization coef- 
ficients greater than those in the initial encoding. That is, the bit rate can be reduced by re-encoding the objects of the 
image in Fig. 46A by rough quantization. 

40 [0273] The selector 21 35 switches a data path such that the combinations of the objects 2304 and 2305, the objects 
2306 and 2307 and the objects 2308 and 2309 are inputted the integration processor 21 36, under the control of the pro- 
file controller 2130. 

[0274] The detailed construction of the integration processor 2136 is the same as that of the object integrator 1 1 23 
of the seventh embodiment in Fig. 34. Accordingly, explanations of the construction and processing of the integration 

45 processor 21 36 will be omitted. 

[0275] The code data on synthesized color image information and code data on mask information are inputted via a 
selector 2137 into the multiplexer 2114. and are multiplexed to code data of one object. The result of the system code 
synthesizer 2131 , the header generated by the header processor 2132, and code data corresponding to the header are 
sequentially inputted via the selector 2137 into the multiplexer 2114. and multiplexed and outputted. 

so [0276] Fig. 57 shows the data structure of code data outputted from the image editing unit 2208 of the eleventh 
embodiment. In Fig. 57. in the video object data, the newly-set PLI code (PLIN-1 in Fig. 57) is provided at the head. 
Then the VO data A-1 -1 corresponding to the background object 2300, and the VO data A-1 -23 corresponding to the 
object synthesized from the objects 2302 and 2304 follow. Further, the VO data B-1 -1 corresponding to the background 
object 2301 , the VO data B-1 -23 corresponding to the object synthesized from the objects 2306 and 2307. the VO data 

55 C-1 -1 corresponding to the object 2302. the VO data D-1 -1 corresponding to the background object 2303. and the VO 
data D-1 -23 corresponding to the object synthesized from the objects 2308 and 2309 follow. That is. seven video 
objects exist in one Visual Object. 

[0277] The code data obtained as above is stored into the storage device 2209. or decoded by the decoder 22 1 0 and 



26 




* EP0 954 181 A2 



displayed as an image as shown in Fig. 48 on the display unit 221 1. 

[0278] As described above, according to the eleventh embodiment, when code data of different profiles and levels are 
synthesized, prof 3e and level are re-defined, and further, the number of objects and the bit rate can be regulated. Thus, 
code data of profile and level desired by the user can be obtained. 
5 [0279] ' Further, respective objects within an image can be arbitrarily synthesized by integrating objects based on the 
relation among the objects (nodes) described in the system code. That is, a synthesizing procedure closer to a user's 
intuitive synthesizing procedure can be realized. 

Modification of Eleventh Embodiment 



[0280] Fig. 55 is a block diagram showing a modified construction of the code length regulator 21 34 according to the 
eleventh embodiment. If input video code data has been motion compensated, a Huffman decoder 2143 decodes the 
quantization DCT coefficients. The Huffman decoder 2143 inputs the obtained quantization DCT coefficients into a high 
frequency eliminator 2144. to eliminate high frequency components by replacing the high frequency components with 

is "0". Then, a Huffman encoder 2143 encodes the output from the high frequency eliminator 2144. That is, the code 
length can be reduced by eliminating high frequency components of the object and re-encoding the data. 
[0281] Fig. 56 is a block diagram showing another modified construction of the code length regulator 2134. If input 
video code data has been motion compensated, the Huffman decoder 2143 decodes the quantization DCT coefficients. 
Then, an inverse quantizer 2146 performs inverse quantization on the obtained quantization DCT coefficients, then a 

so quantizer 2147 quantizes the obtained DCT coefficients using quantization coefficients greater than those used in the 
initial coding. Then the Huffman encoder 2145 encodes the data. That is. the code length can be reduced by decoding 
code data of a motion compensated object and re-encoding the data with rough quantization. 

[0282] Note that in the eleventh embodiment, objects to be integrated may be selected using information unique to 
the respective objects, described independently of the user data or code data, in addition to the relation among the 
25 objects indicated by nodes. That is. objects having similar attributes ("people", "profession" and the like) may be inte- 
grated. Further, the objects 2307 and 2306 may be integrated based on the attributes indicating actions of "people" 
objects such as "giving an injection" and "taking an injection" as selection conditions. 

[0283] Further, objects to be integrated may be selected by the combination of plural conditions such as object size, 
code length, location relation and user's instruction. 
30 [0284] Further, in the eleventh embodiment, objects are integrated based on the relation among the objects (nodes) 
described in the system code, however, the number of object may be reduced by deleting objects selected based on 
the nodes. In this case, the bit rate can be reduced at the same time. 

[0285] Note that the arrangement of the system code memories 2101 to 2104 and the video code memories 2105 to 
2108 is not limited to that in Fig. 47, but more code memories may be provided or one memory may be divided into a 

35 plurality of areas. Further, a storage medium such as a magnetic disk may be employed. 

[0286] As described above, according to the tenth and eleventh embodiments, one code data based on a predeter- 
mined standard can he obtained by synthesizing a plurality of code data, encoded for a plurality of image information 
(objects). Further, the synthesized code data may de decoded by a decoder of arbitrary coding specifications. Further, 
the number of objects and the code length of the code data can be regulated. 

40 [0287] Further, in the above-described respective embodiments, the object 0 is background, however, the object 0 is 
not limited to the background but may be a moving image of a general object or the like. 

[0288] The present invention can be applied to a system constituted by a plurality of devices (e.g., host computer, 
interface, reader, printer) or to an apparatus comprising a single device (e.g., copy machine, facsimile). 
[0289] Further, the object of the present invention can be also achieved by providing a storage medium storing pro- 
45 gram codes for performing the aforesaid processes to a system or an apparatus, reading the program codes with a 
computer (e.g., CPU, MPU) of the system or apparatus from the storage medium, then executing the program. 
[0290] In this case, the program codes read from the storage medium realize the functions according to the embodi- 
ments, and the storage medium storing the program codes constitutes the invention. 

[0291] Further, the storage medium, such as a floppy dsk, a hard cSsk. an optical disk, a magneto-optical disk, CD- 
so ROM, CD-R, a magnetic tape, a non-volatile type memory card, and ROM can be used for providing the program codes. 
[0292] Furthermore, besides aforesaid functions according to the above embodiments are realized by executing the 
program codes which are read by a computer, the present invention includes a case where an OS (operating system) 
or the like working on the computer performs a part or entire processes in accordance with designations of the program 
codes and realizes functions according to the above embodiments. 
55 [0293] Furthermore, the present invention also includes a case where, after the program codes read from the storage 
medium are written in a function expansion card which is inserted into the computer or in a memory provided in a func- 
tion expansion unit which is connected to the computer, CPU or the like contained in the function expansion card or unit 
performs a part or entire process in accordance with designations of the program codes and realizes functions of the 
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above embodiments. 

[0294] The present invention is not limited to the above embodiments and various changes and modifications can be 
made within the spirit and scope of the present invention. Therefore, to appraise the public of the scope of the present 
invention, the following claims are made. 

Claims 



1 . A data processing apparatus having decoding means for decoding code encoded in image object units, said appa- 
ratus comprising: 

10 

detection means for detecting a number of objects included in input code and a number of objects decodable 
by said decoding means; and 

control means for controlling the number of objects of the input code, based on the number of objects and the 
number of decodable objects detected by said detection means. 

75 

2. The apparatus according to claim 1 , wherein if said number of objects is greater than said number of decodable 
objects, said control means reduces the number of objects included in said code to said number of decodable 
objects. 



20 3. The apparatus according to claim 1, further comprising: 



extraction means for extracting location information of the objects included in said code; and 

combining means for combining code of a plurality of objects, based on an instruction from said control means 

and the location information extracted by said extraction means. 

25 

4. The apparatus according to claim 3, wherein said combining means combines code of a plurality of objects away 
from each other by a distance therebetween, wherein said distance being shorter than other distances between 
objects calculated from said location information. 



30 5. The apparatus according to claim 1, further comprising: 

extraction means for extracting motion information indicative of motions of the objects included in said code; 
and 

combining means for combining a plurality of objects based on an instruction from said control means and the 
35 motion information extracted by said extraction means. 

6. The apparatus according to claim 5. wherein said combining means combines code of a plurality of objects having 
the motion information similar to each other. 

40 7. The apparatus according to claim 1, further comprising: 



extraction means for extracting code lengths of the objects included in said code; and 

combining means for combining a plurality of objects based on an instruction from said control means and the 

code lengths extracted by said extraction means. 

45 

8. The apparatus according to claim 7, wherein said combining means combines code of a plurality of objects having 
code lengths shorter than other code lengths. 



9. The apparatus according to claim 1, further comprising initialization means for determining a coding method for 
so encoding the input coda in frame units, and initializing said control means based on the result of determination. 

1 0. The apparatus according to claim 9, wherein said initialization means determines whether said code is encoded 
based on interframe correlation or encoded based on intraframe information. 



55 11. The apparatus according to claim 10, wherein if said code is encoded based on the intraframe information, said ini- 
tialization means initializes said control means. 

12. The apparatus according to claim 1, wherein said code is code of a still image. 
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1 3. The apparatus according to claim 1 , wherein said code is code of a moving image. 

14. A data processing method for decoding code encoded in image object units, said method comprising the steps of: 

5 detecting the number of objects included in input code and the number of objects decodable by said means; 

and 

controlling the number of objects of the input code, based on the number of objects and the number of deco- 
dable objects detected at said detection step. 

io 15. The method according to claim 14. wherein at said control step, if said number of objects is greater than said 
number of decodable objects, the number of objects included in said code is reduced to said number of decodable 
objects. 

16. A computer program product comprising a computer readable medium having computer program code, for execut- 
es ing data processing which decodes code encoded in image object units, said product comprising: 

detecting procedure code for detecting a number of objects included in input code and a number of decodable 
objects; and 

controlling procedure code for controlling the number of objects of the input code, based on the number of 
20 objects and the number of decodable objects detected in said detection procedure. 

17. A data processing apparatus for processing a data array to reproduce an image with a plurality of coded image 
objects, said apparatus comprising: 

25 detection means for detecting a number of image objects included in said data array; and 

control means for controlling the number of image objects included in said data array based on the number of 
image objects detected by said detection means. 



30 



18. The apparatus according to claim 17, wherein 

if said number of image objects is greater than a predetermined number, said control means reduces the 
number of image objects included in said data array. 

19. The apparatus according to claim 18, wherein said predetermined number is a number of objects which can be 
35 processed by decoding means for decoding said data array. 

20. The apparatus according to claim 18, wherein said control means reduces the number of image objects by deleting 
an image object. 

40 21. The apparatus according to claim 20, wherein said control means obtains code lengths of the respective image 
objects in said data array, and deletes the image object based on the obtained code lengths. 

22. The apparatus according to claim 21 , wherein said control means deletes sequentially from an image object having 
the shortest code length. 

45 

23. The apparatus according to claim 20, wherein said control means obtains image sizes of the respective image 
objects in said data array, and deletes the image object based on the obtained image sizes. 

24. The apparatus according to claim 23, wherein said control means deletes sequentially from an image object having 
so the minimum size. 

25. The apparatus according to claim 20. further comprising setting means for setting a priority order of the image 
objects in said data array. 

wherein said control means deletes the image object based on the priority order set by said setting means. 



55 



26. The apparatus according to claim 20. wherein said control means reduces the number of the image objects by inte- 
grating a plurality of image objects. 
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wherein said control means integrates the plurality of image objects selected by said selection means. 

* 28. The apparatus according to daim 27, wherein said control means comprises decodina mean.; far rico^r™ «, . 

3^1c y eCOd ' n9 meanS - and cod,ng means ,or encodin 9 an image object synthesized bv said 

synthesizing means, so as to integrate the plurality ot image objects selected by said selectfon means 

w 29. The apparatus according to claim 28, wherein said control means further comprises counting means for countino 
code lengths of the plurality ot image objects selected by said selection means 9 

ing means. Where ' n ^ ™*" S ^ parameters based «• the results of counting by said count- 

" l^ZTj'T^ '"i^ ? Wherei " 8aU COmr01 meanS ""V™ 5 "Potion means for separating the 
« ma 3« Ejects selected by said selection means into color information and mask information colo in or 
matron s y nth es ,z,ng means for synthesizing the color information separated by said separation ^ans mask n o - 
ma on synthesizing means for synthesizing the mask information separated by saw serration means ^ 

20 ™h hrrr eanS to '. mAt ^ the synthesized by said color Ln^J^S^l 

20 and the mask informal™ synthesized by said mask information synthesizing means. s ^ lnes,a "3 

31 ' irn'ageS" aCC ° rdin9 * ^ M " SeleClion means «*« —"a! selection of a plural of 

25 Il^nr? 5 aC , C ° rdin9 ,0 C ' aim Wherei " Said Se,ection means selecte a Plurality of image objects based on 
spatial location information of the respective image objects. 

33 " 10 °' aim Wherei " meanS -* 3 "™* * objects, one of 

" ^ r^i!^ , ? ara,U L a ? 0reii,,9 10 Claim 27 ■ Wh6rein 58111 selection means * ,ains in *3e sizes of the respective image 
objects in said data array, and selects the image object based on the obtained image sizes. 9 

45 

3a to 0,aim 37, ,ur,her comprisin9 setting means ,or setfn9 a priori » * *• 

wherein said selection means selects the image object based on the priority order set by said setting means. 

" 40 ' Itend a a^ ratUS aCCOrdin9 10 C ' aim 17 ' Wherei " $aU data array iS eod8 data ada P ted to ° r based « *e MPEG4 

41 ' ohf^e^l 9 ^ 61 ^ fof processin 9 a tote array to reproduce an image with a plurality of coded image 
objects, sad method comprising the steps of: 9 

detecting a number of image objects included in said data array and 

*SZS?J£S££ 5? objecte incluc,ed in da,a ^ based on *• ' - ima9e -** 
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42. A computer program product comprising a computer readable medium having computer program code, for execut- 
ing data processing which processes a data array to reproduce an image with a plurality of coded image objects, 
said product comprising: 

£ detection procedure code for detecting a number of image objects included in said data array; and 

control procedure code for controlling the number of image objects included in said data array based on the 
number of image objects detected in said detection procedure. 

43. A data processing apparatus comprising: 

TO 

input means for inputting a plurality of image data to construct one frame, wherein said image data respectively 
including N image objects, where N>1 holds; and 

generation means for generating image data having M image objects, where M>1 holds, constructing said one 
frame, by integrating at least a part of said N image objects based on additional information indicative of rela- 
is tion among the image objects. 

44. The apparatus according to claim 43. wherein said M image objects are an appropriate number of image objects 
to be processed in accordance with a predetermined coding standard. 

20 45. The apparatus according to claim 44, wherein if the number of image objects included in said image data inputted 
by said input means is greater than the number of objects defined by said predetermined coding standard, said 
generation means performs integration processing. 

46. The apparatus according to claim 43, wherein said image data inputted by said input means is adapted to or based 
25 on the MPEG4 standard. 

47. A data processing method comprising the steps of: 

inputting a plurality of image data to construct one frame, wherein said image data respectively including N 
30 image objects, where N>1 holds; and 

generating image data having M image objects, where M£1 holds, constructing said one frame, by integrating 
at least a part of said N image objects based on additional information indicative of relation among the image 
objects. 

35 48. A computer program product comprising a computer readable medium having computer program code, for execut- 
ing data processing, said product comprising: 

input procedure code for inputting a plurality of image data to construct one frame, wherein said image data 
respectively including N image objects, where Nsi holds; and 
40 generation procedure code for generating image data having M image objects, where M£1 holds, constructing 

said one frame, by integrating at least a part of said N image objects based on additional information indicative 
of relation among the image objects. 

49. A data processing apparatus for processing a data array to reproduce one frame image with a plurality of coded 
45 image objects, said apparatus comprising: 

input means for inputting a plurality of data arrays; 

instruction means for instructing synthesizing of a plurality of data arrays inputted by said input means; 
designation means for designating coding specifications of a processed data array; 
so control means for controlling information amounts of the plurality of data arrays inputted by said input means, 

based on the coding specif ications designated by said designation means; and 

synthesizing means for synthesizing the plurality of data arrays with information amounts controlled by said 
control means, based on the coding specifications designated by said designation means. 

55 50. The apparatus according to claim 49. wherein said synthesizing means sets coding specifications of synthesized 
data array to the coding specifications designated by said designation means. 

51 . The apparatus according to claim 49. wherein said instruction means instructs synthesizing including change of 
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spatial locations of the image objects included in said data array. 

.natedby said Cession means. sa« contro, «^SrCaSffi£ 

^ ^atton^aTO^the°ima^e'ot^^ts ^lud^^irs^^^ta'aTay 63113 Se ' eC ' S ima 9 e objscts to be integrated, based on 
55 ' matioT afa,US aCC ° rdin9 10 daim 54 ' Wherei " ,he re,ation —9 ^ '™ ge objects is represented by node .n fo , 



70 



15 



30 



35 



M ' s^aT 3 '" 3 aCCOrd,n9 '° C ' aim 49 ' Wherein * COdin9 ^^ons are adapted to or based on the MPEG4 

45 

65. The apparatus according to daim 49. wherein said data array is adapted to or based on the MPEG4 standard, 
"objet^ 
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inputting a plurality of data arrays; 

instructing synthesizing of a plurality of data arrays inputted at said input step- 
designating coding specifications of a processed data array 
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67. A computer program product comprising a computer readable medium having computer program code, for execut- 
ing data processing which processes a data array to reproduce one frame image with a plurality of coded image 
objects, said product comprising; 

£ input procedure code for inputting a plurality of data arrays; 

instruction procedure code for instructing synthesizing of a plurality of data arrays inputted in said input proce- 
dure; 

designation procedure code for designating coding specifications of a processed data array; 
control procedure code for controlling information amounts of the plurality of data arrays inputted in said input 
io procedure, based on the coding specifications designated in said designation procedure; and 

synthesizing procedure code for synthesizing the plurality of data arrays with information amounts controlled in 
said control procedure, based on the coding specifications designated in said designation procedure. 
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